D3-friendly outputs and templates about twarc

[edsu]

This all sounds good to me @pbinkley so please send the pull requests. FWIW I've half considered creating a new project (twarc-report?) that is oriented around generating a cohesive static report (using d3, bootstrap, etc). If you want to run in that direction I would be willing to help, and it might give you space to think?

[pbinkley]

I've put in PR #42 for the first item.
I wondered about a new project as well. For my purposes, I think the changes will manifest themselves as a series of utilities like directed.py, which analyze a given twitter file and build a dict or array of values, and then call a method in d3json.py to serialize the values in a format that can be dropped into a given D3 example. They'll also have an HTML template file for immediate viewing. To create a new project would mean moving some of what is currently in twarc/utils over, but that wouldn't be a big deal.
For myself, I'm happy to work in a new project or within the existing one. I wouldn't want to go too far without getting you to look at what I've done, if you have the time, since I'm still learning Python as I go.

[ruebot]

twarc-report 👍

[pbinkley]

You can see where I'm going in my timebar branch, which is almost ready for a PR: https://github.com/pbinkley/twarc/tree/timebar . I've got utils/profile.py to do line-by-line processing of tweets and gather basic stats about the tweet set. Then you extend that in a specific context, in this case in utils/timebar.py, which adds timezone conversion and aggregation of values. Finally, d3json.py (which needs to be renamed - it does csv as well) produces output in different formats. I'm renaming my two D3 html outputs to d3directed and d3times and moved their descriptions to the bottom of the readme.
If this pattern is ok, where would you draw the line between twarc and twarc-report? Where does profiler belong?

[edsu]

This is all super @pbinkley So it sounds like you are leaning towards creating a new github repository for twarc-report? If you do I think you want to make twarc-report stand alone as long as you have some data you collected with twarc (line oriented json) you can play. Does this make sense?

[pbinkley]

Yes, I think that makes sense. I'd like to transfer ownership of it to @edsu when my current round is done, though, since your long-term commitment to twarc is likely to be more consistent than mine. Does that sound ok? I intend to finish off the D3 timeline bar graph and the co-tag directed graph, and come up with examples of using the data outputs with other D3 examples out there. I'd like to develop the profiler and the D3 data outputs far enough to support these uses, but I'm sure there will be more that others will want to add eventually.

[edsu]

Well, you could transfer ownership -- but I like the idea of you retaining ownership. I can always help maintain it, and the great thing about github is that if issues don't get addressed and people care they can easily fork it right?

[pbinkley]

OK, OK - we'll try that and see how she goes. Without the edsu brand,
though, it won't get the same level of trust.

Peter Binkley
Digital Initiatives Technology Librarian
Information Technology Services
[email protected]

2-10K Cameron Library
University of Alberta
Edmonton, Alberta
Canada T6G 2J8

phone [phone]
fax [phone]

On 16 January 2015 at 08:29, Ed Summers [email protected] wrote:

Well, you could transfer ownership -- but I like the idea of you retaining
ownership. I can always help maintain it, and the great thing about github
is that if issues don't get addressed and people care they can easily fork
it right?

—
Reply to this email directly or view it on GitHub
#41 (comment).

[ruebot]

What if there was a 'twarc' Github org?

[pbinkley]

I've set up https://github.com/pbinkley/twarc-report and copied the current
D3 examples over. They still need work, but the examples work.

An org would work. Let's see how things go for now.

Peter Binkley
Digital Initiatives Technology Librarian
Information Technology Services
[email protected]

2-10K Cameron Library
University of Alberta
Edmonton, Alberta
Canada T6G 2J8

phone [phone]
fax [phone]

On 17 January 2015 at 05:10, Nick Ruest [email protected] wrote:

What if there was a 'twarc' Github org?

—
Reply to this email directly or view it on GitHub
#41 (comment).