Archive for the 'filesystem' Category
mobifs eifs fmfs

I wrote to the fuse mailing list, about the eifs/fmfs/mobifs. I am still not sure which name fits better, for simplicity I will refer to it as mobifs. Based on the feedback from the mailing list, it became clear for me, that mobifs can be implemented with help of fuse. However implementing it in kernel will avoid double context switch, copying buffers, etc. One of the powerful features of fuse is that you may develop proof of concepts for any kind of filesystem, and the if performance is an issue, recode a part to fit into the kernel. The user space solution for mobifs will be straightforward to implement, at a later stage a kernel module may come into being.

As summary, the goal of mobifs (everything is filesystem) is to allow transparent (read/write) access to the tree structure of files that have such a tree structure under the form of traditional directory tree structure and to present the leaves in form of text, that can be edited without compromising the format expected by the original tools that handle these files.

The driving motivation behind is also to avoid to learn hundreds of command line parameters of hundreds of tools that access such files and to be able to access the data that hides behind with traditional viewers, editors or other tools.

For example an sqllite database file has such a directory structure. instead of issuing sql statements, one would be able to acces such a file like:

#>vi blogs.sqllite@sqllite/tables/blogs/rows/by/uniqueid/mobifs-eifs-story/content

in this example “blogs” is the table containing the blog entries, “by” means that I want to access data by column name, “mobifs-eifs-story” is the unique id of the blog entry, and “content” is the field containing the content.

if your blog database would be on a mysql database on a remote server, you would do sthg. similar:

#>vi blogs.db.id@mysql/tables/blogs/wors/by/uniqueid/mobifs-eifs-story/content

in this case blogs.db.id would contain necessary info how to access the database (server ip, db name, username, password)

eifs/fmfs (everything is filesystem/file mapping filesystem)

I am doing some proof of concept for a virtual filesystem extension. I wrote the following

email to the linux filesystem mailing list.

Hello mailing list members,

I am not sure about the best name, but I am sure it is sthg. very usefull. So I will call it mobifs, hope you do not mind :) I am also not sure if there is sthg. like that out there, but based on my 15 years of linux experience I never met sthg. like what I would like to talk about.
What I would like to have on my linux boxes is to access data the following way:

#>cat /dev/sda1@/partition1/etc/inittab@/defaultranlevel #usecase 0

#>cat /data/xmlfiles/apsettings.xml@xml/main/entry1/subentry/value #usecase 1

#>cat /data/xmlfiles/apsettings.xml@/main/entry1/subentry/value #note @ instead of @xml #usecase 2
#>cat /data/images/linux.img/@/partition1/etc/dhcpd/dhcpd.conf@group/next-server#note twice the @  #usecase 3

I think you got the point. It is actually not a real filesystem, but an extension to the virtual filesystem.How would it work? If the virtual filesystem subsystem of the kernel finds an @ it would redirect the call to the mobifs main module. The mobifs will check if there is any string after @ till the next /. If yes, it will look if there is any driver installed with that name. In usecase 1, it will look if there is a
driver to read xml files. If yes, it will call the driver and pass the rest of the string to it. The xml driver
will interpret the path request and output the necessary entry. If there is nothing specified like in usecase0 (note annoying tmp. mount not needed!!!)mobifs will try to guess, and if he finds a matching filetype (like the file command or mount does) it would pass the path to the driver if there is any driver registered for that. Drivers can be partition readers (do not know oficial technical name
:)), existing filesystem drivers etc. It should be possible to have kerenel drivers and userspace drivers.
It seems rather logical that for reading filesystems one would use kernelspace drivers, and
for trivial file mapping operations userspace program would fit better.

Usecas0 has twice @. This should mean double “redirection” or piping. Mobifs should help to feed the output of one driver with the input of next one, etc. In case of usecase 0, the one driver would read the filesystem on sda1, and feed the read of the driver that is able to read inittab files.

If sthg. like that would become to being, I would see its evolution as 2 main steps: first create
read only support, and then add write support. The same would happen with the drivers…

I was thinking about this the last few days, started to hack around the kernel, but before going
to the wrong direction, I wanted to ask more experienced people about what and what not to do!

for some more imagined usecases, see the PS.

rgrds,
mobi phil

P.S.

#>cat /databases/mysqldatabase/db.sql@mysql/tables/table1/2345/rowname
#>cat /databases/mysqldatabase/db.sql@mysql/tables/table1/rowname/1232
#>cat /databases/mysqldatabase/db.sql@mysql/tables/table1/rowname

#>cat /xxx/xx/xx/xx/elfimage@elf/symbols

#>cat /sourcecode/cpp/thismodule/thatfile.h@cpp/methodist
#>cat /sourcecode/cpp/thismodule/thatfile.h@cpp/includelist
#>cat /sourcecode/cpp/thismodule/thatfile.cpp@cpp/methods/thatmethod

#>cat /sourcecode/cpp/thismodule/thatfile.h@cpp/methodist

#>tail -f /var/log/messages@grep/error #maybe this is less usefull

Linux loopback device partitions

Linux kernel has powerful features, however sometimes it can be disappointing when you discover that trivial details do not work or do not work the very obvious way. Such an example is the loop back device. You may mount a file as a loop back device, but you cannot access its partition as “sub-devices” like it happens with the physical discs. There are workarounds by mounting the partition knowing the start and length of it, but that might be a bit risky.

another measurement of the bottlnecks of openmoko

trying to copy with dd from /dev/urandom to the flash card it takes 4 seconds, if there is no framebuffer activity. If just doing some resizes during the copy, the times are in range of 15 to 30 seconds.

wikipediafs, proposal for improvement

I tried to improve a bit the wikipediafs python script, but there are some small basic design mistakes. Beside those, it is impossible to have a list of all the article. Therefore I would use the api.php like

http://<YOURPEDIA>/api.php?action=query&list=allpages&apfrom=B&format=json

I would then precreate (by the driver) a directories called _allfiles, _search. Under the _allfiles I would then precreate directory names (A, B, C … Z, or if there would be too many articles under one directory AA, AB, … BA, BB… ZZ), and would use the above mentioned script to populate these directories with articles beginning with the corresponding name.

wikipediafs works for me

However it is stated on the homepage of wikipediafs that the project is not maintained, it works! The only one change one has to make is:

user.py:56:        self.login_page = “%s?title=Special:Userlogin” % self.basename

into

user.py:56:        self.login_page = “%s?title=Special:UserLogin” % self.basename

(note the change from Userlogin to UserLogin)…

and now it is straightforward and easy to edit the wiki with vim… I will write probably a plugin to jump to links automatically. I wish one day there will be fuse fore wince :) and I will be able to edit wikis with vim on mobile :)

Please note that wikipediafs is a userspace filesystem and is mounted with help of http://fuse.sourceforge.net.

versioning filesystem

I was thinking that there should be something that imitates clearcase on linux, some sort of versioning filesystem. It turned out that there are few projects in this direction. I found mainly 2 of them:

The filesystem implementations have similarities and differences. Wayback and copyfs are implemented in userspace and they use the services of any kernelspaced filesystem to implement the versioning. Ext3cow is a kernel space implementation and it relies on ext3.

Whatever the others do, I am thinking maybe to write a very trivial versioning filesystem, that could use the services of any other filesystem (ext3, jfs, xfs etc).  Basic ideas:

  • versions (or diffs) would be stored in the same directory as the normal file, but when mounted as versioned filesystem, the version files would not be visible
  • to be continued
fuse everywhere, even on openmoko

I did not check if by default the fuse is built in the openmoko kernel, but I came today to the conclusion that I missed a lot of oportunities by not using fuse. I think on an embedded device this could create some interesting opportunities. Even more I was investigation dbus performance, and came to the conclusion that a lot was dbus does, could be done by fuse in a more efficient way…

Something like http://www.runtimeaccess.com/projects/rta/index.html could help programs to publish there data in a filesystem… etc…

Anyway the idea to map databases you use in everyday life to the filesystem, and do things like find/grep index etc on them, is really powerful!!!

I will do all this from now on…

even more… I will mount even things like wikipedia… dictionaries etc. :)

I hope it will not mean security issues:)

Anyway, the most exciting are:

Fuse is a way the prolongation of the ideas like reiserfs…

 

 

 

 

nfs4 is very slow

After months of fight and frustration with nfs4 on Ubuntu, I switched to nfs3. And finally I can see the  light at the end of the tunnel! As I was describing here, I made some measurement about copy speed of small files towards nfs server and back. Copying to the server was almost 10 times slower with nfs4 than copying from the server (same harddisk type both on server and client).

Now with nfs3 the speeds are the same in both directions!!!

json makes it fast…

Another ingredient for fast programs is JSON. The JSON format is a concurrent of XML and YAML. The later stays closer to JSON. I found so far two libraries for C++ that can parse and write JSON structures:

  • http://jsoncpp.sourceforge.net/
  • http://blog.beef.de/projects/tinyjson

… there are even further few listed on the homepage json.org

… There is even a query language jaql