That's what I've been doing, but I wouldn't call it clever:sbs wrote:Will some clever programming the boxscore range should be able to resolve all of the problems from the last names only in the play-by-play.
I can't promise I have time to do it, but I bet EvanZ either has already done it or is doing it right now.colts18 wrote:No problem. Will you be able to parse out the pbp data from 97-00?
What does the parsing out process entail? I have no knowledge on this so I want to know. How long does it usually take to parse out this data?kpascual wrote:I can't promise I have time to do it, but I bet EvanZ either has already done it or is doing it right now.colts18 wrote:No problem. Will you be able to parse out the pbp data from 97-00?
Haha. I'm now definitely considering it.kpascual wrote:
I can't promise I have time to do it, but I bet EvanZ either has already done it or is doing it right now.
The raw, parsed files could be hosted elsewhere (but you were referring to NBAWOWY?)EvanZ wrote:Haha. I'm now definitely considering it.kpascual wrote:
I can't promise I have time to do it, but I bet EvanZ either has already done it or is doing it right now.
It would be great for my site to have all the data going back that far. Although, it could get rather expensive to host what would probably be about 10 GB worth of pbp and matchup data.
EvanZ wrote:Haha. I'm now definitely considering it.kpascual wrote:
I can't promise I have time to do it, but I bet EvanZ either has already done it or is doing it right now.
It would be great for my site to have all the data going back that far. Although, it could get rather expensive to host what would probably be about 10 GB worth of pbp and matchup data.
Yes, there is some kind of code. The primary challenge of parsing play-by-play is determining who is on the court. Using the NBC dataset, I've got an error rate that is very, very low. It's almost a perfect process with very little manual correction involved. Every day it takes me about 5 minutes to update the site. The one good thing about going back and dealing with old data, is that once you do it, you don't have to mess with it again. I'd love to add it to my site, but like Ken said, it's a matter of finding time.How does this parsing out process work? Is there some kind of code to make it work?
How long does it take to parse out a season's worth of pbp data? Are you able to do an APM on that data?EvanZ wrote:@Daniel, the way my site works is that the app is running off Nodejitsu and the database is running off MongoLab. The plan I'm on right now is 1 GB per month storage and that's about $10/mo. Not a big deal. But it goes up steeply from there. 4 GB per month is $40/mo. Every GB after that is another $11, so 10 GB would be over $100/mo. Since I don't make money off the site, that's $1200/yr coming out of my pocket. And for all I know the database might end up being around 15 GB to go back to '97.
@colts18, right now nbawowy is just this season. My play-by-play is coming from NBC Sports which was the cleanest, easiest source to parse that I could find (for example, providing full names on every play). Unfortunately, before I could scrape prior seasons from the site (which I think they had at one point), they took them all down (I'm assuming at the request of the NBA).
Yes, there is some kind of code. The primary challenge of parsing play-by-play is determining who is on the court. Using the NBC dataset, I've got an error rate that is very, very low. It's almost a perfect process with very little manual correction involved. Every day it takes me about 5 minutes to update the site. The one good thing about going back and dealing with old data, is that once you do it, you don't have to mess with it again. I'd love to add it to my site, but like Ken said, it's a matter of finding time.How does this parsing out process work? Is there some kind of code to make it work?
Hey, Ken. There's a D3 meetup in SF tonight at Trulia. Any chance you're going? I'll be there.
The short answer is it doesn't hardly take any time at all once you've written the code. The longer answer is it takes a lot of time to write the code.colts18 wrote:
How long does it take to parse out a season's worth of pbp data? Are you able to do an APM on that data?
Dammit, I missed it. I was signed up to go, but forgot I had a rec league basketball game. But yeah we should actually hang out sometime. Sloan conference? Other meetups?EvanZ wrote: Hey, Ken. There's a D3 meetup in SF tonight at Trulia. Any chance you're going? I'll be there.
This. The primary constraint isn't time, it's usually effort.EvanZ wrote:The short answer is it doesn't hardly take any time at all once you've written the code. The longer answer is it takes a lot of time to write the code.colts18 wrote:
How long does it take to parse out a season's worth of pbp data? Are you able to do an APM on that data?
And yes, once you do that, you can calculate APM or RAPM or whatever.
Do you think you would be able to do it? Or is it too hard?EvanZ wrote:The short answer is it doesn't hardly take any time at all once you've written the code. The longer answer is it takes a lot of time to write the code.colts18 wrote:
How long does it take to parse out a season's worth of pbp data? Are you able to do an APM on that data?
And yes, once you do that, you can calculate APM or RAPM or whatever.
How long would it take to do that for the 4 seasons? If its not too long, and someone taught me how to do it, I guess I could try.J.E. wrote:If someone converts the data into text files with the format
gameid TAB linenumber TAB time TAB [team_id] description
I can take a crack at it using the parser I wrote for bbr PBP
(I don't want to do the html-> text conversion because, as I said, I think at some point the PBP will appear on bbr, for which I already have a crawler/converter)