New filesystem project

ioquake3 dev
21

I added a new feature that lets you customize which files are added to the download and pure lists when running a server. You can specify (or remove) both individual paks and categories of paks from either list by modifying the fs_download_manifest and fs_pure_manifest cvars.

Some of the cases where this can be useful:

  • To remove excess mod paks from the download list that clients don’t really require
  • To add resource paks to the download list that clients do require but aren’t marked as referenced by the normal mechanism
  • To trim the pure list to use the current map instead of everything in baseq3, which enables running a valid pure server with thousands of pk3s installed without overflows

Additional documentation on this feature is available here. If you would like to try it on your server you’ll probably need to build the project from source, unless you use Windows in which case you can get a test build from the release page. Let me know if you have any questions or comments.

1 Like
22

I was wondering if it would be okay if I created a pull request for this project on GitHub. I think the project is now stable enough to officially propose for ioquake3 inclusion, and a pull request would provide a place for people to discuss it in that context.

If you haven’t been following this project, here’s an overview of the links to check it out. There have been a lot of improvements since the original release last year. I’d love to hear any feedback.

1 Like
23

Hi Chomenor, maybe you remember, I’m a long time tester from the very beginning of your project and I still use it for my programming experiments, you did a really great job. I never had any problems during this period, it really works well, impressive project, imo.
One question, can you explain a bit more if enabling downloads are safer now? Iirc, somewhere here I read that noone should enable auto-downloading (security risk), so is this REALLY fixed with your project?

It would be cool if your work would be included into ioquake3, especially because it doesn’t break mod compatibility as far as I know.
Though, someone said you have to maintain/update your code regulary, and thats a reason why such big projects are usally not included (though, on the other hand, other ‘projects’ had a bigger influence and others are even no longer maintained and these ‘projects’ were included nevertheless…(Mumble, OpenAL, etc.))

So, I hope your project will find its way into ioquake3!
Anyways, please stay with ioquake3, I really like your work (even if I don’t understand it, codewise).

Well done!

P.S.: as aready noted, your code formatting is a bit ‘strange’ :wink:

24

Automatic downloads should be a lot safer if the download directory restrictions here are enabled. It can block, or restricted to trusted hashes only, the qvm files that are the source of most security vulnerabilities. These options aren’t enabled by default currently, but that could be changed later, especially if cl_allowDownload were to be enabled by default as well.

I didn’t consider this project complete when I first released it, so it needed more maintenance initially. It shouldn’t require too much maintenance now, besides minor tweaks here and there, like any project. I recommend evaluating this project by testing it in real conditions, like download-enabled servers and with existing map and mod packs. It should be pretty stable now, but let me know if you do notice any compatibility or other issues.

I think my code formatting is kind of Python-like, where the indentation is the main guide, there are minimal extra lines, and the braces are just added in afterwards where needed. That’s something that can be changed very easily by script though, if it was requested for ioq3 inclusion.

Thanks for the support and testing!

25

My personal setup with my ioquake3 fork, Spearmint, involved 40 game directories for 30 unique standalone game releases (retail/demo/etc) with more waiting to be added (granted most are not playable beyond loading maps). 260 pk3 files. 16 pk3dirs for separating data I made for testing a single thing.

Sort of agree, conceptually:

  • Save downloads into a separate directory with a option to disable loading qvms from loaded pk3s. I don’t agree with the sorting precedence or keeping downloaded pk3s loaded when you required by server. However, apparently deep integration with the launcher to allow/deny is planned instead.
  • A way to specific which pk3s to load would be useful for hosting servers and local testing. Loosely similar to new filesystem’s “fs_download_manifest” / “fs_pure_manifest”. Though, I would probably vote for a literal list of pk3 names to be used in the order listed (single cvar for referenced and pure pk3 lists).

The changes I listed above would not require a complete rewrite of the filesystem. I don’t really want any of the other features listed in the new filesystem readme to be added to ioquake3.

Don’t agree:

  • Searching all game directories (to fix broken mods?) is strange and goes against the Quake 3 design. Also I have other standalone games in my search path. I don’t want Quake 3 content pulled into OpenArena or other separate games, if that makes sense.
  • Sharing renderer settings across mods would be nice but it doesn’t seem apporiate to change the config behavior. (Don’t worry we can have consistent video resolution with the lancher!)
  • … I don’t want to go through listing reasons for all of them.

The issues listed by SmileTheory earlier in the thread still apply as well. I’m opposed to merging new filesystem and realistically there isn’t a way you can change ‘new filesystem’ to make it something I would want to merge into ioquake3. It doesn’t really fit with the general concept I work under of ‘make the smallest possible change to fix the problem’. Yeah, it may not always happen.

26

Thanks for the reply. I’m sorry you don’t have a good impression of my project. I have a few questions, or requests for clarification, if you don’t mind:

  • You said you don’t agree with the “sorting precedence”. Is this in relation to the download directory feature, and what aspect of the precedence do you mean specifically?

  • My download/pure manifest feature can be used to load a literal list of pk3 names, it just supports additional set keywords on top of that. Are you saying would prefer to just remove the set keywords and allow pk3 names only? Why do that, and wouldn’t it just make the feature a lot less flexible and more difficult to use?

The reason this project is done as a rewrite of the filesystem is not to support the supplemental features in the readme, but rather to make architectural changes necessary to support large numbers (up to thousands) of pk3s with good performance and stability. The file precedence has been overhauled to make it almost impossible for any pk3 in baseq3 to override the core paks and break the game or other maps, which means you can download maps and mods from servers or bulk packs without worrying about conflicts. The caching and optimizations allow fast load times with thousands of pk3s installed and the new debug commands show exactly which resources are loaded and how they are selected. The debug commands are pretty great, IMO, if you haven’t tried them yet. I personally wouldn’t want to go back to not having them for any kind of development use.

  • As the readme states, the full inactive mod support is just used as a last resort if the alternative is a texture error. Is there a specific practical reason why this is bad? The only argument I’ve heard is that it could theoretically confuse map authors. At any rate this is entirely cvar controlled and can be disabled or removed for ioq3 integration purposes.

  • The restricted inactive mod support (setting=1, missionpack and pure list pk3s only) allows maps designed for team arena to work with non-team arena mods without convoluted extra download workarounds that servers have to do now. It also gives advanced pure server admins some tools to do things like submodding, if they are targetting only new filesystem clients (like a standalone game scenario). Do you know of any problem that would arise from this feature?

  • I know the shared config file behavior is debatable. That’s why I designed it from the start to be cvar controlled and easily removed if it’s a condition for ioq3 integration.

If there is any way we could discuss your concerns and maybe keep possible integration of this project on the table, that would be great. Otherwise I’ll drop the project here, even though it’s a bit sad and I don’t think this project quite got the consideration it deserves… that’s the way it goes.

Good luck with ioquake3 and your future plans and thanks for the response. Maybe I’ll be able to contribute in some other way in the future :slightly_smiling_face:

27

I added pk3dir support now. It seems to work better than the original implementation because it supports case-insensitive access like a pk3, even on Linux, although I haven’t played around with it too much yet. @zturtleman perhaps you’d like to give it a try on your config, and see how it works for you? You could also try the find/compare commands with it.

28

Again, I would like to ask if the ioquake3 maintainers would mind if I open a pull request for this project? It would be nice to be able to at least make the case for this project before you reject it. It would also allow this project to be listed and discussed along with other proposed contributions. There may be code or ideas that are selectively useful to ioquake3 or other projects, even if the project itself isn’t accepted.

If you would like to close the thread on this forum so there isn’t duplicate discussion, that would be fine.

Thanks

29

Seems a good idea to me.
I had some time ago so many PK3 that Quake3 was really overcrowded.
Many bugs, and loading problems.

By the way it’s still unclear for me how is the game supposed to manage maaaaaany PK3, when inside of them there is similar named files.
For example I load a map within a link to a texture blabla.tga
and in another pk3, scanned and loaded on game launch, I already have another blabla.tga texture file,
wich of them is used in the map?
The 1st scanned-one, the last-scanned one (wich kick the previous one in the memory load?)

I already asked about that problem in another forum and nobody ever could explain me…

So I think a better filesystem that can manage better all that stuff would probably help yeah!!!

30

Thanks for the comment!

In the simple case, within the same mod directory (baseq3 for example), the blabla.tga from the alphabetically higher pk3 file will get used. So if ztextures.pk3 and ytextures.pk3 both contain a blabla.tga, the one from ztextures.pk3 is used. If there is an active mod (fs_game) active then pk3s in the mod directory will override the ones in baseq3.

In this project there are some extra conditions, such as that the ID paks and the map pk3 take precedence over other arbitrary pk3s in baseq3. The idea is that maps load in a “bubble” where they load like they are running on a clean install with just their own pk3 and the ID paks, and only when assets are missing from there will other paks be used according to the alphabetical policy.

It’s important to note that in practice maps don’t directly reference a tga file, they reference a generic type that is mapped by the engine to either a shader, tga, jpg, png, or other image type. The original filesystem uses a very sloppy approach in which shaders always override tgas, which always override jpgs, etc. and shaders are compared to other shaders using weird conventions that depend on the name of the .shader file within the pk3. This project makes a lot of improvements in this area and in most cases the precedence works the same regardless of whether the assets are of different types. For example a jpg in the map pk3/ID paks/current mod will still override a shader/tga with the same name in some other arbitrary pk3.

I hope that answers your question. Let me know if you want me to go into more detail about anything, or if you want specifics about how files are stored in memory etc.

31

Interesting, I’ll give It a try when some time.
My current approach, was to explode all PK3, and compil them one by one in a unique filesystem.
Means, I’m having only one shader file, one botfile, etc etc.
And then check manually for files with a name identical, and then modify one, with the referencies to this file too in any file possible of each new map, new bot, etc etc…
many work :slight_smile:
but I ain’t found else.
I just add what I want, as I want, and make It much more efficient (no duplicates, so save memory? load time at list?)
Much time to choose what is worth or not too (many maps sometimes are cute, but without fun, or ugly and no fun at all…)
Many PK3 give the exact same textures from one to another map, and the exe has to load all this at startup.
While he’s limited in memory…
The main problem was, when putting many many many PK3 in the baseQ3, some where not loading because no more memory available.
I don’t wanna manually call/load my maps by console, nor change my files in directory each time I wanna have a fun game with mates.
By the way I also try to adapt some boats, remap them If ugly too much (I have few few skill) make them more tuff in some abilities (botfiles) or make the red/blue version avaible for CTF when not made.

I’m not sure I won’t have memory problem either that way (with too many maps added, sometimes just browsing maplist pictures to choose one, I can get after some time to get a error, and no more mappictures displays, then game need to be restart.

So definately will try your new FS and see. :slight_smile:
In anycase, It’s awesome to see Quake3 is going Forward, with still many meticulous passionate people!
For me It’s still the best…

32

What are the advantages to having the fs_game concept? What if instead there was the following alternative:

  1. All mods used the top of the homepath and basepath directories

  2. Servers would explicitly communicate which initial pk3 files were required.

  3. Clients would be able to explicitly specify which pk3 files would load (if permitted with unpure and semipure servers).

  4. Loaded pk3 files could include dependency files that would explicitly specify other pk3 files to load.

  5. pk3 files not indicate under either of the above 3 attributes don’t load.

Perhaps that approach could even be backwards compatible with existing servers, but dependcy files would have to be created for existing maps when needed. There could still be something like and fs_game for servers, just as a convenient way of indicating required pk3 files by dropping them in the folder, and that kind of fs_game info could still be communicated for the old clients. But for both the clients and the servers, most other things would be in the top of the homepath (like demos, configs, screenshots, qkey, etc).

I’m not saying that design would definitely be the way to go, but at least on the surface, I think that it would be a very simple design, easy to use, would probably be backwards compatible, would ensure that pk3 files that are undesired to load don’t load, and allow setups to have an unlimited number of pk3 files without issues.

33

@Frenchguy Even with this filesystem there is still a limit on the number of maps that will show up in the menu. That requires changing the UI code which I haven’t done yet. This project does fix a lot of issues when running a lot of pk3s but just not the map menu limit.

Note that if you modify bots or other files that are part of the ID paks you will probably need to put them in basemod instead of baseq3 for them to work with this project.

34

@dGr8LookinSparky The current file structure with baseq3 and separate mod dirs is what people already have on their systems, and the way existing mods are packaged, so it doesn’t seem practical for ioquake3 to make significant changes.

Could you clarify which issues with the existing system you are trying to solve? This project does already have some of the characteristics you described, like supporting a practically unlimited number of pk3s and preventing undesired pk3s from causing problems. The only change it makes to the file structure is requiring “permanent” mod pk3s to be moved from baseq3 to basemod.

35

I didn’t get to look over your system and try it out yet, but I will definitely at some point. My primary interest with a new file system is for another game that is based on ioquake3 (as one of the project leaders of GrangerHub working on Tremulous), so the goals may be different from that of the goals of the ioquake3 project. Although for awhile we have been working to keep backwards compatibility with old servers/clients, we will probably change that requirement (at least with old clients), so we could have less restrictions for a new file system than ioquake3.

But yes, one of the main issues is being able to not be limited in the amount of pk3 files you have downloaded. I have found that many players don’t even know how to get to their homepath to delete excess pk3 files (although if there was a button from the main menu for opening your homepath in a file explorer, that could help a lot), and if you have a development server where you update new pk3 files often, after awhile you have to change the fs_game or many people would crash from just trying to connect.

Other issues are with conflicts, commonly used settings changing with fs_games, and the possibility of the main menu being overridden by a mod in an undesirable way (we did change our file system for the client a bit to use a common fs_game for the configs/screenshots/etc, and force the client to only load our ui and menus on startup so that it can’t be overidden, but in some ways that is a broken approach, like you can’t play back demos nor load mods from the main menu).

It sound like your project might address the above issues. Again, I didn’t get a chance to review your work yet, so I’m in no position to argue one way or another the merits of your project’s design and implementation at this time. It is possible that using your file system may be a beneficial route for our project.

With that said, another important issue is simplicity, and I have been wondering lately if the fs_game concept is just fundamentally flawed making things more complicated than they have to be, and perhaps might not even be necessary for backwards compatibility.

There was a another project that implemented a file system close to what I described in my previous post, although it wasn’t designed to for backwards compatibility (but perhaps could be). The other thing is that that project is licensed GPLv3+, and I don’t think that ioquake3 will upgrade their GPLv2+ license anytime soon. But we will probably upgrade to GPLv3+ for our project at some point, and I was thinking of porting and adapting the FS of that other project. But perhaps there might be some inspiration from that file system for improving your project, and if you implement the same kind of features, you may not have to upgrade your GPLv2+ license (if you don’t want to), provided your code is original and non-derivative.

If you’d like, I can om you information about that other file system.

36

Ah, that makes a lot more sense given it is for a separate project rather than ioquake3 itself.

My project definitely takes care of the the number of pk3 files limit, both the hard limit and the performance aspect. This is one of the main reasons I did this project originally, because I have Elite Force servers with a lot of maps in which people are constantly hitting the 500 pak limit and I wanted a solution for it that didn’t just raise the limit, but allowed the game to stay fast and stable over that number of paks as well.

There is indeed support for a common settings file, and that option is currently enabled by default in my repository.

The way this project currently works as it is designed for ioquake3 is that the ID paks are identified by hash and given special precedence over other pk3s in baseq3, so that random pk3s in baseq3 can’t override the core game behavior. Users can manually move pk3s to a different folder called basemod to override the ID paks without loading a mod, and servers aren’t allowed to download to that location.

The special hash handling is required for Quake 3 because the ID paks are stored in baseq3 along with random other paks. If this project were to be implemented in Tremulous or other standalone games, where you have control over where your core paks are saved, you would probably move them to basemod or a similar special folder outside basegame and then the filesystem can just prioritize that folder without needing to store a list of hashes.

I don’t think the fs_game concept is necessarily fundamentally flawed. There are issues with authenticity and versioning, especially when mod paks are downloaded from servers - how do you tell which paks are the “correct” ones for a mod? This is a problem with any mod-support setup though, not specific to fs_game, and there are possible workarounds like hardcoding a list of “correct” pak hashes for a given fs_game value in the engine, or if you really want to get into it a way for mod authors to embed a signed manifest into their mod paks to allow updates. Pure servers generally avoid these issues, and if they are targeting clients using this filesystem they can be semi-pure as well which helps for admins who don’t want to restrict stuff like player models.

I skimmed the filesystem article you sent. It seems interesting. It has different goals from this project, because this project is intended to be compatible with ioquake3 and that project isn’t. I can’t say which approach I would recommend for you, without knowing much about either Tremulous or that filesystem. The regular fs_game model is probably simpler and more familiar to users, whereas that project probably has other advantages like more fine-tuned control over mods and the ability for different versions of mods to functionally coexist while sharing some of the same paks.

If you are interested in trying to port my filesystem your project, I’d be happy to help with that. Looking at your files.cpp here, if that is the correct file, it doesn’t look too different from ioquake3’s files.c, so I’m optimistic it wouldn’t be too hard to do at least a quick and dirty port.

1 Like
37

I very much appreciate your offer to help us port your file system to our project, thank you @Chomenor :slight_smile: ! It does look like your file system project might have potential for our project. I’ll discuss with the other GrangerHub developers about if your file system would be the best fit for our project, and we’ll get back to you. Regarding our current file system, it is very similar to ioquake3’s file system, with the main difference being that we have multiprotocol support specifically for the 3 protocols that Tremulous uses.

38

Thanks for the consideration. If you and the other developers are interested, I wouldn’t mind doing a prototype port, even if you aren’t committed to using it yet. It would probably take me a few days and be a good testing and learning opportunity. I’ve heard about Tremulous a lot but never actually played it, so it would be interesting to try.

This project is very flexible so I don’t think the protocol stuff will be any issue at all.

Feel free to contact me if you or have any questions!

1 Like
39

Awesome, thank you! Sure, if you wouldn’t mind putting in the time to port your file system to our project as a prototype @Chomenor , we would be very interested in trying it out, that would be an excellent way for us to evaluate your file system for our project :slight_smile: .

40

UH!

was whecking for any news as I wanted to finally give a try…
…just realized you only gave the raw sources,
I already checked the Cygwin stuff and how to compilate the ioquake3 exe…
I’m not dev at all…
even with full rpocedure I don’t get ho to add the packagesdespite I find them online…

Why don’t you share your already compiled x86 x64 self versions?
any link womewhere?

thanx