I asked this once in the main SR3R thread, but it was off-topic and easily missed, so I guess I'll repeat here.
There used to be a bit of talk about a fanproject of making a large index to cross-reference all the SR3 rulebooks, and give an index to the ones without.
Does anyone know if this was ever done, or even attempted?
I've seen a number of comments spread through different threads, but never anything come out of it.
Full Version: Shadowrun 3 Index
whoah, whoah, whoah. easy to find? cross-referencing? YOU'RE KILLING THE SPIRIT OF SHADOWRUN!
If this is an effort to start this as a com-pro, I'll help. The index should be for SR 1,2 & 3 though, now that they won't be surprising us with any new books it can be a complete reference.
Is there software for this sort of thing? How would one go about putting it together?
Is there software for this sort of thing? How would one go about putting it together?
| QUOTE |
| Is there software for this sort of thing? How would one go about putting it together? |
It would be better to make it online and searchable.
Something like this would be easy to implement in MySQL/PHP. The hard part would be collecting the data.
Wouldn't it be cool if people could edit the list on the fly, ala wikipedia?
I think putting together this project, at least setting up the MySQL/PHP framework, would take no more than 8 hours. It wouldn't be especially pretty, but there's always time to tidy up the framework's output html later.
Who would be interested in entering data for such a project?
MySQL...YourSQL whatever it takes 
Ideally one could search online but a printable super index would be good as well. I'd do data entry if it were relatively non tedious ie page by page rather than individual entries, I'm not sure how the software operates.
My thoughts were to scan, OCR or copy indexes (and perhaps contents) from all books and sort. Those with official pdfs could easily submit a copy. I have already gathered half a dozen in thinking how this might be done.
Ideally one could search online but a printable super index would be good as well. I'd do data entry if it were relatively non tedious ie page by page rather than individual entries, I'm not sure how the software operates.
My thoughts were to scan, OCR or copy indexes (and perhaps contents) from all books and sort. Those with official pdfs could easily submit a copy. I have already gathered half a dozen in thinking how this might be done.
Using PDFs makes it legally questionable - either for scanning pages or for pulling pages out of a published PDF.
I think it comes down to three possibilities. PHP, a wiki, or just using HTML with links between categories.
If there's already a Shadowrun-specific wiki out there, that would probably be the simplest way to do it, especially for group contributions. HTML, on the other hand, would also be very fast and simple, but might not look as good, and would be a bit tougher to edit.
I don't know PHP, but I've been told MySQL is similar to MS Access, which I have used. If so, that would probably function best, but again probably be difficult, and retain the lack of open editing.
I think it comes down to three possibilities. PHP, a wiki, or just using HTML with links between categories.
If there's already a Shadowrun-specific wiki out there, that would probably be the simplest way to do it, especially for group contributions. HTML, on the other hand, would also be very fast and simple, but might not look as good, and would be a bit tougher to edit.
I don't know PHP, but I've been told MySQL is similar to MS Access, which I have used. If so, that would probably function best, but again probably be difficult, and retain the lack of open editing.
| QUOTE (Sahandrian) |
| Using PDFs makes it legally questionable - either for scanning pages or for pulling pages out of a published PDF. |
To clarify, when I said contents I meant the book's contents page(s) which are usually given out with the first chapter as a preview of new books.
As for (US) legalities, would this not qualify as 'fair use'? Think of the compiled equipment tables, history collections etc on the SR Archive site for example. I reckon we go ahead if we have interest and deal with any complaints later, especially since I hail from a jurisdiction far, far away
A lot of the equipment has already been compiled. Look at Man and Machine for cyberware, bioware and chemicals. Cannon for weapons. Matrix for decking and computer gear ...
I was thinking of the ultimate SR gear list as an expression of fair use rather than something we would make for the index.
I just noticed the other day that you sent a PM about 2 years ago regarding MIJI rules - still want 'em? Am I the only one around who never checks PM's
I just noticed the other day that you sent a PM about 2 years ago regarding MIJI rules - still want 'em? Am I the only one around who never checks PM's
Plus you can't search a scanned PDF.
| QUOTE |
| MySQL...YourSQL whatever it takes |
A wiki would work fine - but I believe it would be possible to create a better interface for this specific project. Wikis are great at community-edited topics, what is being discussed is a community-edited book index.
| QUOTE (Link @ Jun 7 2007, 09:08 PM) |
| Ideally one could search online but a printable super index would be good as well. I'd do data entry if it were relatively non tedious ie page by page rather than individual entries, I'm not sure how the software operates. My thoughts were to scan, OCR or copy indexes (and perhaps contents) from all books and sort. Those with official pdfs could easily submit a copy. I have already gathered half a dozen in thinking how this might be done. |
Since a wiki would have the perfect back-end for this, would it be possible to just modify the display/edit interfaces for mediawiki and call it a day? MediaWiki is PHP/MySQL, anyone know how complex this task would be?
The biggest advantage of a wiki would be that there's a relatively low learning curve for the contributers. In any sizable volunteer project you've got to consider that factor or watch your project die a slow agonizing death. On the other hand, given that the material won't be expanding at a terribly huge rate if you can keep interest high to start then you should be able to get the initial input in easily...
Unfortunately a wiki is almost too broad for what is called for in this sort of project - and certainly doesn't have a way to store the background relationship information for easy index - if you wanted to make something simple and searchable for just the information you need in an index - you would need to make a custom application. I could see the following tables in it:
keywords: id, word - id is autoincremented key
books: id, name, abbreviation, edition_number - id is autoincremented key
index: keyword_id, book_id, page_number - all three values are key
You probably would want the end-user application to have a page where you can download a pdf of a master document. And a page to search through the index for specific keywords, or to list words found in specific books. I would suggest having the php page generate the list of indexes out in either html or pdf - searcher's choice (there's enough php classes out there to convert html to pdf quickly, that this feature is almost negligible for a decent programmer). If the only information stored is index and keyword then there's no reason to do any cross-linking.
An OCR of content pages could certainly make for a good start to any indexing project, and it may be worthwhile to write a quick input page to take in these files, but that can be a tricky deal depending on the format of the file. You can also have a page that allows users to enter the data by hand, but that can be tedious, though it's not like tedium isn't a built in part of this sort of thing. Open community editing on the fly is actually the easier of the options than restricted editing (since restricted editing would require someone to write in a login and password system as well).
But all of that assumes you have a programmer and a hosting site that allows you use of php and mysql (geocities freebie accounts need not apply) - so if you don't - you're pretty much stuck with what's out there.
In which case, wiki it is. In that case you may want to approach it less like an index (boring to read topics) and more like an encyclopedia. Each topic page has a *very* brief overlook - and then cites references at the end of the entry by book, edition and page number. A little more indirect, but sans programming something custom - it's what you've got. You won't be able to easily print it out in any sensical format.
As for how complex it would be to modify MediaWiki - at this point I would say: Infinitely complex. You haven't really defined what you *want* the interface to be. Without a target, no programmer could even begin to take a shot.
Unfortunately a wiki is almost too broad for what is called for in this sort of project - and certainly doesn't have a way to store the background relationship information for easy index - if you wanted to make something simple and searchable for just the information you need in an index - you would need to make a custom application. I could see the following tables in it:
keywords: id, word - id is autoincremented key
books: id, name, abbreviation, edition_number - id is autoincremented key
index: keyword_id, book_id, page_number - all three values are key
You probably would want the end-user application to have a page where you can download a pdf of a master document. And a page to search through the index for specific keywords, or to list words found in specific books. I would suggest having the php page generate the list of indexes out in either html or pdf - searcher's choice (there's enough php classes out there to convert html to pdf quickly, that this feature is almost negligible for a decent programmer). If the only information stored is index and keyword then there's no reason to do any cross-linking.
An OCR of content pages could certainly make for a good start to any indexing project, and it may be worthwhile to write a quick input page to take in these files, but that can be a tricky deal depending on the format of the file. You can also have a page that allows users to enter the data by hand, but that can be tedious, though it's not like tedium isn't a built in part of this sort of thing. Open community editing on the fly is actually the easier of the options than restricted editing (since restricted editing would require someone to write in a login and password system as well).
But all of that assumes you have a programmer and a hosting site that allows you use of php and mysql (geocities freebie accounts need not apply) - so if you don't - you're pretty much stuck with what's out there.
In which case, wiki it is. In that case you may want to approach it less like an index (boring to read topics) and more like an encyclopedia. Each topic page has a *very* brief overlook - and then cites references at the end of the entry by book, edition and page number. A little more indirect, but sans programming something custom - it's what you've got. You won't be able to easily print it out in any sensical format.
As for how complex it would be to modify MediaWiki - at this point I would say: Infinitely complex. You haven't really defined what you *want* the interface to be. Without a target, no programmer could even begin to take a shot.
To help those who have proficiency with various applications discussed here to decide what course to take, here is a copy of the index from the Corporate Security Handbook using OCR. The ideal platform would let us enter data in this format as it is not impractical to collect indexes from the few dozen SR books.
Discuss
Discuss
[ Spoiler ]
Hm... looks like it's pulling the information in as a tab delineated list - which means it can be translated *quite* well by php's built in functions ...
*sigh* *hangs head and feels guilty* Me and my taking on of projects...
Ok - someone find some hosting...
I'm not sure if the folks here at dumpshock would be up for it or not but it's worth asking. I can probably crunch an initial version out within a day or so. I need to wrap up a project for a job first is all.
*sigh* *hangs head and feels guilty* Me and my taking on of projects...
Ok - someone find some hosting...
I'm not sure if the folks here at dumpshock would be up for it or not but it's worth asking. I can probably crunch an initial version out within a day or so. I need to wrap up a project for a job first is all.
I originally intended to try and do it myself, if it didn't exist, but at this point it's gone beyond my ability, at least in the programming area. Sometimes I wish I'd stuck with my original CS major (and not just because I'm in school an extra year due to the switch).
I could still try and type up some of the indexes, at least if there are some books I have that those with access to a scanner don't.
And what about the books without indexes? Work one out ourselves?
Edit: Oh. And I'm no use for hosting. I just have Geocities.
I could still try and type up some of the indexes, at least if there are some books I have that those with access to a scanner don't.
And what about the books without indexes? Work one out ourselves?
Edit: Oh. And I'm no use for hosting. I just have Geocities.
When compiling the Corporate Security Handbook index I put a csh entry on each row after the page reference(s) for sorting purposes.
eg. Access control 28-29 csh
As for hosting, my ISP provides 10mb homepage allowance (is this what you're talking about?)
eg. Access control 28-29 csh
As for hosting, my ISP provides 10mb homepage allowance (is this what you're talking about?)
Hey, as per usual, I should be able to host anything you need. my little webserver gets pretty good speed and I'm not likely to run out of space any time soon. So, if you'd like I'm sure we can get sometime set up.
So for the books that don't have an index, I was thinking two things. First, go through and note whenever any of the words in the other indexes are used =\ yeah annoying, i know. And also add other words that might be useful.
to do this... it would probably be best to convert one of the books to text and then search (ya know, with a search function) through it and note whenever any specific words are used.
Or, if you get a notepad and start typing as you read through the book and on each line just put the keyword you find, a tab, and the page number, and keep then in no sort of order, I'm sure we (myself if you what) could whip up a little script to take that text file and alphabetize it and make an index from it.
Also, I can probably cobble together a DB to store the index, and after that a search script or a complete index would be rather simple to make. all that's needed for the index table is:
id, word, source_book, and page number
And then make the words able to be in there more than once, and then just pull them out all at the same time. But yeah, I'm up for helping if anyone wants.
So for the books that don't have an index, I was thinking two things. First, go through and note whenever any of the words in the other indexes are used =\ yeah annoying, i know. And also add other words that might be useful.
to do this... it would probably be best to convert one of the books to text and then search (ya know, with a search function) through it and note whenever any specific words are used.
Or, if you get a notepad and start typing as you read through the book and on each line just put the keyword you find, a tab, and the page number, and keep then in no sort of order, I'm sure we (myself if you what) could whip up a little script to take that text file and alphabetize it and make an index from it.
Also, I can probably cobble together a DB to store the index, and after that a search script or a complete index would be rather simple to make. all that's needed for the index table is:
id, word, source_book, and page number
And then make the words able to be in there more than once, and then just pull them out all at the same time. But yeah, I'm up for helping if anyone wants.
I suspect chibu you would find in practice that your table design won't be as efficient as you would like: you'll end up repeating words in different books, as well as books for each word, and cross referencing between books and words could be a major CPU drain.
What I was thinking for the books that don't already have an index is unfortunately it would have to be by hand by someone. If someone can create a CSV or tab delineated file out of Excel then they could upload it all in one chunk - it's basically the same thing as the file created by OCR...
How would we be able to retain the page number if the book were converted to text file? Just have the person doing the converting make note of it?... oh. Idea! This would require some familiarity with PDFs and Word though but it would be muuuuch faster if someone did it this way...
Ok - this assumes that whatever PDF you're working with has text or has been scanned with OCR to create a PDF with text, and that you have a full copy of Adobe Acrobat... You *should* then be able to export the file out to Word to create a .doc file with the exact same page numbering. MS Word has the ability to take a set of index words in a separate file and auto create an index. (Look it up in Word's help files to learn how.) So that would allow you to just create an index out of thin air without spending hours on hours reading through it. It would just need a quick once over to be sure it got everything. Then - *that* file can be pulled back into PDF for conversion via OCR or have the created index copied and pasted into Excel to create your tab delineated file for uploading.
Unfortunately if it's a book that currently has no PDF - then the old fashioned way will probably have to do. I can't think of a way to make that easier.
What I was thinking for the books that don't already have an index is unfortunately it would have to be by hand by someone. If someone can create a CSV or tab delineated file out of Excel then they could upload it all in one chunk - it's basically the same thing as the file created by OCR...
How would we be able to retain the page number if the book were converted to text file? Just have the person doing the converting make note of it?... oh. Idea! This would require some familiarity with PDFs and Word though but it would be muuuuch faster if someone did it this way...
Ok - this assumes that whatever PDF you're working with has text or has been scanned with OCR to create a PDF with text, and that you have a full copy of Adobe Acrobat... You *should* then be able to export the file out to Word to create a .doc file with the exact same page numbering. MS Word has the ability to take a set of index words in a separate file and auto create an index. (Look it up in Word's help files to learn how.) So that would allow you to just create an index out of thin air without spending hours on hours reading through it. It would just need a quick once over to be sure it got everything. Then - *that* file can be pulled back into PDF for conversion via OCR or have the created index copied and pasted into Excel to create your tab delineated file for uploading.
Unfortunately if it's a book that currently has no PDF - then the old fashioned way will probably have to do. I can't think of a way to make that easier.
There are plenty of books with indices/indexes and for those that don't hopefully some of them are suitable pdf's.
I retrieved the list of SR books from the rpg site for reference. We can work out what we have and put out a list of those indexes someone may be able to supply.
Just to clarify, is the index document format noted above (eg. Access control 28-29 csh - ad nauseum) suitable? I'd do a few more books if so.
I retrieved the list of SR books from the rpg site for reference. We can work out what we have and put out a list of those indexes someone may be able to supply.
[ Spoiler ]
Just to clarify, is the index document format noted above (eg. Access control 28-29 csh - ad nauseum) suitable? I'd do a few more books if so.
Yes.
So long as it is a regular repeating pattern I can get a php script to recognize it. In this case it's 'keywords' followed by a tab or two, followed by 'page or pages separated by a dash', and the line ends with a standard breakline so... anything that doesn't match that pattern can be ignored.
I would not suggest adding the abbreviation for the book t the end of that line - just use what OCR provided. The page that'll ask for the file to be uploaded will ask you what book it came from. That should save you some typing.
So long as it is a regular repeating pattern I can get a php script to recognize it. In this case it's 'keywords' followed by a tab or two, followed by 'page or pages separated by a dash', and the line ends with a standard breakline so... anything that doesn't match that pattern can be ignored.
I would not suggest adding the abbreviation for the book t the end of that line - just use what OCR provided. The page that'll ask for the file to be uploaded will ask you what book it came from. That should save you some typing.
And for the record, i do know that the book names (or rather numbers) and words would be repeated alot, but SQL DBs are actually much more efficient than you give them credit for. It can do a million line query in under a second, so it shouldn't be a problem.
One can convert a book from hardcopy to scanned imaged to ORC text of the book in the original format. I did it with Fields of fire once. (unless whoever's reading this is concerned with the copyright protection of the book, in which case, i have done no such thing ^-^) I do not, however know of being able to turn a pdf into a word doc. I'll see what i can come up with though.
One can convert a book from hardcopy to scanned imaged to ORC text of the book in the original format. I did it with Fields of fire once. (unless whoever's reading this is concerned with the copyright protection of the book, in which case, i have done no such thing ^-^) I do not, however know of being able to turn a pdf into a word doc. I'll see what i can come up with though.
It's (at least in the copy of Adobe I'm looking at here) under the 'save as' options... I'll double check to make sure that's not some peculiar setup on my system.
It's been a long time since I last posted but the code is wrapped up and going through final bug testing. I want to add administrative controls in version two before opening it up to anycomers. Yay, for getting a new job and losing your free time... I'm considering some of the hosting requests offered so far (thank you all!) but will address that once the final administrative considerations for the indexer are taken care of.
There is a file like this floating around somewhere. It's basically an equipment list for everything in SR, plus a bunch of converted CP2020 stuff thrown in. It was a fan based project made like 6 years ago or somesuch, so none of the SOTA books. it's an 8 page pdf I found while trolling the internet. I may still have it but, as has been pointed out, it's firstly of dubious legality (since it lists equipment stats for everything in addition to the book reference) and includes a lot of unbalanced CP stuff. Plus, it's equipment-only and out of date to boot.
*boat rocking*
Is it ready yet?
Is it ready yet?
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.