![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
fandom Meta: archiving stories directly from LJ
So, this is all (once again) ![[livejournal.com profile]](https://www.dreamwidth.org/img/external/lj-userinfo.gif)
cesperanza's fault. It all started with a posting she made in which...well, I'll let you read it for yourself if you're interested, but what came to me while reading it was we need better archiving. Not that existing archives aren't beautiful and whatnot, but there is no tie in with LJ, and the LJ community is thriving, and many people (such as me) are posting directly to LJ and not bothering to archive (for a variety of reasons.)
The problem with the LJ community is: stuff gets lost. I discovered that myself when digging up stories for my reclist. There are just too many wonderful gems disappearing into the mists of time. The folks at places like![[livejournal.com profile]](https://www.dreamwidth.org/img/external/lj-community.gif)
ds_weekly and others do a fantastic job of keeping us in touch with each other on a weekly basis, but they aren't searchable...and there is just no guarantee that LJ won't disappear like the wind in a desert.
The Proposed Solution
What I'm proposing is this: I'm willing to write an archiving script that will essentially pull LJ story postings and all the accompanying meta information and stuff them into a searchable database.
It would only do so for willing writerly participants. All that would be required of them is that they friend the archiving LJ users (ds_archivist or ts_archivist to start with) and follow a particular template when posting wherever they usually do, something like this:
Title: Guess Who's Coming to Dinner
Author: arrow00 (must be lj username)
Pairing: F/V
Rating: PG
Summary: Ma Vecchio and Dead!Bob find an uneasy truce.
etc. (TBD)
(Hm. I think I might actually write this story.)
The participants can post to whatever communities they like; I will simply make the archivist users join all the communities involved and check those communities and users daily for new stories.
What do you guys think? Will this fly? Are people willing to have their stories added to a centralized, searchable archive automatically?
The beauty of this system is a user can change their story and have it rearchived if they make their edits and then change the date. Sure, it will result in stories showing up more often (maybe they can preface the story with [posting updated] or something so people won't get mad.)
Obviously, there is more to think about and plan, but I find myself massively excited by this idea: it might bring the power and features of LJ to join with the missing features of an archive.
Heck, we can even (I think) include a comment box on the bottom of the story so folks can send their comments right back to the original LJ post.
Oh, and the best part about this: LJ will have already formatted the story html all pretty. I'll just grab everything between the cut tags.
cesperanza's fault. It all started with a posting she made in which...well, I'll let you read it for yourself if you're interested, but what came to me while reading it was we need better archiving. Not that existing archives aren't beautiful and whatnot, but there is no tie in with LJ, and the LJ community is thriving, and many people (such as me) are posting directly to LJ and not bothering to archive (for a variety of reasons.)The problem with the LJ community is: stuff gets lost. I discovered that myself when digging up stories for my reclist. There are just too many wonderful gems disappearing into the mists of time. The folks at places like
ds_weekly and others do a fantastic job of keeping us in touch with each other on a weekly basis, but they aren't searchable...and there is just no guarantee that LJ won't disappear like the wind in a desert.The Proposed Solution
What I'm proposing is this: I'm willing to write an archiving script that will essentially pull LJ story postings and all the accompanying meta information and stuff them into a searchable database.
It would only do so for willing writerly participants. All that would be required of them is that they friend the archiving LJ users (ds_archivist or ts_archivist to start with) and follow a particular template when posting wherever they usually do, something like this:
Title: Guess Who's Coming to Dinner
Author: arrow00 (must be lj username)
Pairing: F/V
Rating: PG
Summary: Ma Vecchio and Dead!Bob find an uneasy truce.
etc. (TBD)
(Hm. I think I might actually write this story.)
The participants can post to whatever communities they like; I will simply make the archivist users join all the communities involved and check those communities and users daily for new stories.
What do you guys think? Will this fly? Are people willing to have their stories added to a centralized, searchable archive automatically?
The beauty of this system is a user can change their story and have it rearchived if they make their edits and then change the date. Sure, it will result in stories showing up more often (maybe they can preface the story with [posting updated] or something so people won't get mad.)
Obviously, there is more to think about and plan, but I find myself massively excited by this idea: it might bring the power and features of LJ to join with the missing features of an archive.
Heck, we can even (I think) include a comment box on the bottom of the story so folks can send their comments right back to the original LJ post.
Oh, and the best part about this: LJ will have already formatted the story html all pretty. I'll just grab everything between the cut tags.
Re: Nifty Tool
However, I will be running the pilot program off of LJ, and hammering out the kinks there, since LJ has the most users. Other journals would come after the primary functionality had been worked out.
Re: Nifty Tool
The only concern I would have right now would be using the other archives and html... 852 prospect and WWOMB use text format for a reason. They do not except html.
Re: Nifty Tool
852prospect, once you have submitted the files, saves them as .html files on the server. What I propose is to work with the archive server itself to save the stories as html on the server in the same format and stick the meta information in the db.
In the case of WWOMB it's slightly more complicated, because I believe it saves text with boldface and italic tags in the db and then formats the text on output. So, yeah, I'd have to swap out the p tags for line returns, but leave the itals and bold. I'm not sure, too, how to resolve the user issue (WWOMB has actual users who can log in and edit their stories after the fact. So there's the issue of mapping ljuser name to WWOMB username.)