![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
fandom Meta: archiving stories directly from LJ
So, this is all (once again) ![[livejournal.com profile]](https://www.dreamwidth.org/img/external/lj-userinfo.gif)
cesperanza's fault. It all started with a posting she made in which...well, I'll let you read it for yourself if you're interested, but what came to me while reading it was we need better archiving. Not that existing archives aren't beautiful and whatnot, but there is no tie in with LJ, and the LJ community is thriving, and many people (such as me) are posting directly to LJ and not bothering to archive (for a variety of reasons.)
The problem with the LJ community is: stuff gets lost. I discovered that myself when digging up stories for my reclist. There are just too many wonderful gems disappearing into the mists of time. The folks at places like![[livejournal.com profile]](https://www.dreamwidth.org/img/external/lj-community.gif)
ds_weekly and others do a fantastic job of keeping us in touch with each other on a weekly basis, but they aren't searchable...and there is just no guarantee that LJ won't disappear like the wind in a desert.
The Proposed Solution
What I'm proposing is this: I'm willing to write an archiving script that will essentially pull LJ story postings and all the accompanying meta information and stuff them into a searchable database.
It would only do so for willing writerly participants. All that would be required of them is that they friend the archiving LJ users (ds_archivist or ts_archivist to start with) and follow a particular template when posting wherever they usually do, something like this:
Title: Guess Who's Coming to Dinner
Author: arrow00 (must be lj username)
Pairing: F/V
Rating: PG
Summary: Ma Vecchio and Dead!Bob find an uneasy truce.
etc. (TBD)
(Hm. I think I might actually write this story.)
The participants can post to whatever communities they like; I will simply make the archivist users join all the communities involved and check those communities and users daily for new stories.
What do you guys think? Will this fly? Are people willing to have their stories added to a centralized, searchable archive automatically?
The beauty of this system is a user can change their story and have it rearchived if they make their edits and then change the date. Sure, it will result in stories showing up more often (maybe they can preface the story with [posting updated] or something so people won't get mad.)
Obviously, there is more to think about and plan, but I find myself massively excited by this idea: it might bring the power and features of LJ to join with the missing features of an archive.
Heck, we can even (I think) include a comment box on the bottom of the story so folks can send their comments right back to the original LJ post.
Oh, and the best part about this: LJ will have already formatted the story html all pretty. I'll just grab everything between the cut tags.
cesperanza's fault. It all started with a posting she made in which...well, I'll let you read it for yourself if you're interested, but what came to me while reading it was we need better archiving. Not that existing archives aren't beautiful and whatnot, but there is no tie in with LJ, and the LJ community is thriving, and many people (such as me) are posting directly to LJ and not bothering to archive (for a variety of reasons.)The problem with the LJ community is: stuff gets lost. I discovered that myself when digging up stories for my reclist. There are just too many wonderful gems disappearing into the mists of time. The folks at places like
ds_weekly and others do a fantastic job of keeping us in touch with each other on a weekly basis, but they aren't searchable...and there is just no guarantee that LJ won't disappear like the wind in a desert.The Proposed Solution
What I'm proposing is this: I'm willing to write an archiving script that will essentially pull LJ story postings and all the accompanying meta information and stuff them into a searchable database.
It would only do so for willing writerly participants. All that would be required of them is that they friend the archiving LJ users (ds_archivist or ts_archivist to start with) and follow a particular template when posting wherever they usually do, something like this:
Title: Guess Who's Coming to Dinner
Author: arrow00 (must be lj username)
Pairing: F/V
Rating: PG
Summary: Ma Vecchio and Dead!Bob find an uneasy truce.
etc. (TBD)
(Hm. I think I might actually write this story.)
The participants can post to whatever communities they like; I will simply make the archivist users join all the communities involved and check those communities and users daily for new stories.
What do you guys think? Will this fly? Are people willing to have their stories added to a centralized, searchable archive automatically?
The beauty of this system is a user can change their story and have it rearchived if they make their edits and then change the date. Sure, it will result in stories showing up more often (maybe they can preface the story with [posting updated] or something so people won't get mad.)
Obviously, there is more to think about and plan, but I find myself massively excited by this idea: it might bring the power and features of LJ to join with the missing features of an archive.
Heck, we can even (I think) include a comment box on the bottom of the story so folks can send their comments right back to the original LJ post.
Oh, and the best part about this: LJ will have already formatted the story html all pretty. I'll just grab everything between the cut tags.
no subject
no subject
My proposal is orthogonal, anyway. I am hoping we can use existing archives such as 852prospect and squidge's DSA. If OTW eventually comes out with an open source archive package that's more robust than existing archives, we could always migrate the data to a new server and set it up. (I'm hoping for better searching. At present, you can't really search the story text at the aforementioned archives.)
Wherever we do end up, however, will be decided before we start gathering the stories. So anyone who has an objection to the ultimate archive location can simply opt not to participate.
->Arrow (up late with weird stomach-fu)
no subject
What's interesting is that I was thinking the other day about the possibility of doing up a "Gen TS for Slash Lovers" list (a friend and I did one - ending up with 100 stories) for Blakes 7) An archive would be a godsend if I did have a go... I do like lists {g}
no subject
no subject
This is a weird and complicated story, but essentially the people who seem to have the biggest problems with OTW have a personal issue against the folks behind the scenes (to the point where they have done fairly unscrupulous things like outing the project coordinators to RL family and employers) or because the OTW project goals coincide with other individual archive/fan history projects. There seems to be a mentality of "but I'm already doing that!" among OTW's detractors and the sense that OTW is poaching on someone else's territory. Which is...well, crap, basically, and scans to me as a bit petty considering fandom in general needs all of the independence, structural and logistic support it can get. But I don't want to get into trouble discussing the specifics of a situation of which I only have the vaguest of understandings. The goals of OTW are worthy ones, and the conduct of the group behind it has been exemplary - very calm, cool and professional. Which cannot be said of its detractors.
Anyway, as to the archiving question, I think it's a great idea - I say, the more off-LJ archives, the better! OTW is a great project and I fully support it, but it can't hurt to make fic available in a wide variety of places, particularly fandom-specific ones.
Your idea of creating a script to automatically pull LJ entries (which authors can continue to edit and update) and channel them into a searchable database sounds great! I don't have any programming ability but I'd like to offer my help and support.
no subject
>"but I'm already doing that!"
Personally, I'm a true believer in "tools, not rules," which is to say: people go where the tools are, and the best tool (unless there are anti-trust shenanigans like Microsoft engages in) usually wins. LJ, in this case, is the current leader.
And tools tend to inform the community they create. If a tool encourages linking (such as LJ does) people will link. If the structure of the tool forces people into isolated threads (such as LJ does) then it's more difficult to centralize the information. That's why my proposal.
In a way, what I am proposing is in keeping with OTW's precepts. We'll be using LJ's resources for our indexing.
If someone makes a better tool, it might take a while, but I do believe people will start shifting over. I'll try to make my tool flexible enough that it can handle multiple journaling sites (e.g., greatestjournal) and do the parsing based on their respective syntices.
I think I will proceed with creating the various accounts and start working on the script. Even if I can't get writers to participate, I can use it for my personal archiving purposes.
no subject
I am the "some woman" (eeep!)
Your system is something like what I was picturing. I'd like to see if it is possible to only slightly alter existing habits (something like inserting tags or whatnot) and using the existing power of the resources we use.
no subject
Plus, it doesn't have cutie icons. :)
We'll see if I can get any traction with my idea, which I think of as sort of "glue" between the best features of both methods.
no subject
Title:
Author: KarieAuthoress
email: karierauthoress@gmail.com
Permission to archive: Yes, but please tell me where.
Fandom: The Sentinel
Genre:
Pairing:
Summary:
Warnings:
Notes:
For all my stories, I do this.
For TS writers though, I think there is an LJ community who is at least attempting to help us keep track of some of our stories.
I'll keep an eye on this for later...
no subject
I like your template very much. It's what I had in mind. I would have to encourage people to rate their stories in a readable way and also include pairings.
no subject
I agree about the tags, especially on some journal styles where there are no tags to click on and search. That can often be frustrating. Or if a writer doesn't use tags at all.
My only concern is most likely a technical one, something about the wording of your proposal. Clarification would be appreciated. Where would these stories be archived and in what format? I ask for a couple of reasons, but I want to make sure I don't put my foot in my mouth in the asking.
Nifty Tool
You, the writer, post your stories in xxx community LJ. You opt-in and use the posting format. At regular intervals , the script pings the LJ community and gathers those stories from those authors who have opted in and delivers them to the designated archive
Where the stories end up - well that is something that you will know BEFORE you opt-in. The real work, IMHO will be at the back end - with the people setting up and keeping the archives going. And those archives might be all over the place depending on the fandom. Personally, I think having multiple archives makes fandom more robust and resilient and this tool will work wonderfully in that arena. It also makes life simpler for the writers - you just post to the one LJ community and your story is uploaded for you to the acrhive.
arrow00 - let me know if I have any of this backwards.
Re: Nifty Tool
And, yes, participating authors would know the ultimate destination before opting in. The destination would not change without notifying authors and allowing them to "opt out" of having their stories moved.
The format would be html, the same formatting that appears on the writer's lj. I would just grab everything from between the cut tags. So I would require that the writer *use* lj-cut to indicate the start of their story.
I will write up a requirements document once I have ironed out the technical issues. But please feel free to continue to ask any questions you might have, because they reveal areas that need fleshing out.
Re: Nifty Tool
Am I going to have to switch back to using both LJ and IJ in order to continue participation should I decide to "opt in" to this?
Re: Nifty Tool
However, I will be running the pilot program off of LJ, and hammering out the kinks there, since LJ has the most users. Other journals would come after the primary functionality had been worked out.
Re: Nifty Tool
The only concern I would have right now would be using the other archives and html... 852 prospect and WWOMB use text format for a reason. They do not except html.
Re: Nifty Tool
852prospect, once you have submitted the files, saves them as .html files on the server. What I propose is to work with the archive server itself to save the stories as html on the server in the same format and stick the meta information in the db.
In the case of WWOMB it's slightly more complicated, because I believe it saves text with boldface and italic tags in the db and then formats the text on output. So, yeah, I'd have to swap out the p tags for line returns, but leave the itals and bold. I'm not sure, too, how to resolve the user issue (WWOMB has actual users who can log in and edit their stories after the fact. So there's the issue of mapping ljuser name to WWOMB username.)
no subject
no subject
no subject
Laurie
no subject
no subject
no subject
no subject
The rest of this post is blither.
I've just arrived at dS fandom, and I would love to have an LJ-oriented archive. I try to keep current with new fic by following the weekly communities, picking up new names from the discussion and then using that info to search on DSA, but I know I'm missing wonderful stuff.
Consider: this fandom is a decade old. Testimony to the crack that is dS, and the Properly Prepared writers who improve on canon daily. But did someone penning the very first CoTW fic, on 13 May 1998, think I'd want to read it in 2008?
no subject
And blither away.
no subject
As a side note to the community maintainers: I encourage them to use the current LJArchive tool and back up their communities every week. I wish there was an automated tool for that.
no subject
no subject
no subject
no subject
But there would be the possibility of doing this for other fandoms as well; once I have the kinks worked out it shouldn't be too hard to add fandoms to the list.
no subject
*regards arrow sternly from beneath lowered brows*
no subject
Yes, it did occur to me that once I get sunk into this I might not surface for a while. OTOH, I might go back and forth between coding and writing, writing and coding. They exercise two very different parts of my brain. Sometimes I dream code on the screen, and sometimes I dream of boys boys boys.
A girl's gotta have variety. :)
no subject
BTW -- I know I have a bunch of your stuff to catch up on. ::drools (attractively... well, or not):: RL has kept me from being on LJ much for six (freaking) months now, except for sporadic little microbursts here and there, and I have approximately six million three hundred and eighty-two fics and friends' posts to catch up on. I may require IV amphetamines (and RL is still not cooperating as fully as it might), but eventually I intend to get a little less behind than I currently am. (God, I hope so. It's driving me crazy, everything I haven't read. Or said... )
no subject
However, I don't think fandom stuffies should be viewed as obligations, so please don't feel any on my account. I'm just glad you are a-ok. And I'm happy to hear you will participate in my little plan!