gadget file storage with variable maxpacket

Post by ilya
hello,
i guess this question is primarily for alan stern, but i
do not know his e-mail.
i was wondering how hard would it be to implement the
file storage gadget with variable packet size, larger than
512, that could be specified as a module option or better
yet use the maxpacket field in the usb_ep structure.

Then where would the maxpacket field in the usb_ep structure get
initialized in the first place? A module option would be possible.

Post by ilya
this
will be neccessary for wireless usb, among many other
things.

What other things?

Post by ilya
i am not really familiar with writing to disk from
kernel space, so any suggestions would be appreciated.

Changing the high-speed maxpacket size wouldn't be hard. It would
involve editing the block of code that starts with:

if (gadget_is_dualspeed(gadget)) {

in the fsg_bind() routine. It wouldn't affect writing to disk at all.

Alan Stern

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
linux-usb-***@lists.sourceforge.net
To unsubscribe, use the last form field at:
https://lists.sourceforge.net/lists/listinfo/linux-usb-devel

ilya

2007-11-14 18:02:41 UTC

hello alan, thank you for replying.
see my response bellow.

Then where would the maxpacket field in the usb_ep structure get
initialized in the first place? A module option would be possible.

Post by ilya
this
will be neccessary for wireless usb, among many other
things.

What other things?

wireless usb will be introducing a lot of new control requests, flow
control [ie: stalling endpoints should be the last resort because in wusb
you can NAK and the host will wait for a special device notification to send
a transfer token instead of constantly polling as in wired usb], association,
etc. i think a lot of these things could be handled by the controller driver,
but there are a lot of other stuff that could be standardized and handled
by the "gadget" client side.

what does david brownell think about this?

Post by ilya
i am not really familiar with writing to disk from
kernel space, so any suggestions would be appreciated.

Changing the high-speed maxpacket size wouldn't be hard. It would
if (gadget_is_dualspeed(gadget)) {
in the fsg_bind() routine. It wouldn't affect writing to disk at all.

i figured out how to change the maxpacket in the descriptors, what
i am having problem with is queuing-in of the requests and
writing to disk. the fs gadget tends to queue in requests of size 512
during writes/reads of, for example 64k, to either stay in the page
boundaries or because it rounded down to stay in the block boundaries.
i, on the other hand, will be doing transfers of maxpacket 2048 so
now i either have to concatenate transfers in my code, or tweak fs gadget
to queue in requests with sizes of maxpacket multiples. i had
success doing this for reads but not so much for writes.

i think it would be nicer if fs gadget queued in requests based on
the maxpacket specified for each endpoint by the controller driver
and took care of all the page/block boundaries business during the
actual reading/writing to disk.

thank you for your time.
-- ilya

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
linux-usb-***@lists.sourceforge.net
To unsubscribe, use the last form field at:
https://lists.sourceforge.net/lists/listinfo/linux-usb-devel

Alan Stern

2007-11-14 23:36:52 UTC

Post by ilya
wireless usb will be introducing a lot of new control requests, flow
control [ie: stalling endpoints should be the last resort because in wusb
you can NAK and the host will wait for a special device notification to send
a transfer token instead of constantly polling as in wired usb], association,
etc. i think a lot of these things could be handled by the controller driver,
but there are a lot of other stuff that could be standardized and handled
by the "gadget" client side.

Do any of these things affect wired USB?

Post by ilya
i figured out how to change the maxpacket in the descriptors, what
i am having problem with is queuing-in of the requests and
writing to disk. the fs gadget tends to queue in requests of size 512
during writes/reads of, for example 64k, to either stay in the page
boundaries or because it rounded down to stay in the block boundaries.

What do you mean? Are you talking about request lengths for USB
transfers? For disk reads (IN to the host), the request length is the
same as the amount read from the disk, which is limited only by the
buffer size (the default is 16 KB). For disk writes the situation is
similar.

Yes, some requests are smaller in order to align with page boundaries.
But once the alignment has occurred, the following requests should all
be as large as possible.

Post by ilya
i, on the other hand, will be doing transfers of maxpacket 2048 so
now i either have to concatenate transfers in my code, or tweak fs gadget
to queue in requests with sizes of maxpacket multiples. i had
success doing this for reads but not so much for writes.

You don't have to tweak anything. Just because maxpacket is 2048, that
doesn't mean all your transfers have to be 2048 bytes long. There's
nothing wrong with short packets or with transfers that extend across
multiple packets.

(At least, that's true for wired USB. I don't know much about wireless
USB; perhaps it's different.)

Post by ilya
i think it would be nicer if fs gadget queued in requests based on
the maxpacket specified for each endpoint by the controller driver
and took care of all the page/block boundaries business during the
actual reading/writing to disk.

In the end it wouldn't make much difference. Can you provide any
examples showing how it would help more than a little bit?

Alan Stern

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
linux-usb-***@lists.sourceforge.net
To unsubscribe, use the last form field at:
https://lists.sourceforge.net/lists/listinfo/linux-usb-devel

ilya

2007-11-15 16:31:23 UTC

Do any of these things affect wired USB?

not at all, 2.0 is ok. but but wait 'till 3.0, i know it will have
flow control similar
to wusb and variable maxpacket sizes too. although this is long time from now,
so lets worry about wireless first.

What do you mean? Are you talking about request lengths for USB
transfers? For disk reads (IN to the host), the request length is the
same as the amount read from the disk, which is limited only by the
buffer size (the default is 16 KB). For disk writes the situation is
similar.
Yes, some requests are smaller in order to align with page boundaries.
But once the alignment has occurred, the following requests should all
be as large as possible.

i was talking about the lengths of USB transfers and these alignments
are what causing the problem. it was ok for high speed because the
maxpacket is the same as the block size [if i am correct?] but with larger
max packets, and especially with maxpackets of non multiples of 512,
there is a problem. see my response below.

You don't have to tweak anything. Just because maxpacket is 2048, that
doesn't mean all your transfers have to be 2048 bytes long. There's
nothing wrong with short packets or with transfers that extend across
multiple packets.
(At least, that's true for wired USB. I don't know much about wireless
USB; perhaps it's different.)

so if it is a single-packet transfer, then yes, there is obviously nothing
wrong with short packets. i am concerned with transfers that are made
up of multiple packets. with INs, if you blindly send out the data that the
gadget queues-in [and have maxpacket greater then 512] you will end up
sending out a short packet in the middle of this multi-packet transfer and
short packet = end of the transfer. so for this case one would have to
make sure to concatenate any requests that are less than maxpacket in
order not to terminate a transfer prematurely. but now how does the controller
driver know that this request is part of a multi-packet transfer and that
request is actually a single transfer with size of less than maxpacket; it
would have to look at the mass storage commands, which is the file storage
gadget's job. with OUTs it is inconvenient because most hardware use
buffers that are size of, or multiples of, maxpacket; so with these short
packet requests you now have read out a buffer in multiple requests,
which adds overhead and is more prone to errors/problems.

-- ilya

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
linux-usb-***@lists.sourceforge.net
To unsubscribe, use the last form field at:
https://lists.sourceforge.net/lists/listinfo/linux-usb-devel

Alan Stern

2007-11-15 18:06:04 UTC

Post by ilya
so if it is a single-packet transfer, then yes, there is obviously nothing
wrong with short packets. i am concerned with transfers that are made
up of multiple packets. with INs, if you blindly send out the data that the
gadget queues-in [and have maxpacket greater then 512] you will end up
sending out a short packet in the middle of this multi-packet transfer and
short packet = end of the transfer. so for this case one would have to
make sure to concatenate any requests that are less than maxpacket in
order not to terminate a transfer prematurely. but now how does the controller
driver know that this request is part of a multi-packet transfer and that
request is actually a single transfer with size of less than maxpacket; it
would have to look at the mass storage commands, which is the file storage
gadget's job. with OUTs it is inconvenient because most hardware use
buffers that are size of, or multiples of, maxpacket; so with these short
packet requests you now have read out a buffer in multiple requests,
which adds overhead and is more prone to errors/problems.

Ah, now I understand your point. Yes, the driver does contain an
implicit assumption that the maxpacket value evenly divides the sector
size.

We can fix the problem easily enough. Just take out the code in
do_read() and do_write() that handles partial pages, and let the VFS or
block layer worry about reorganizing the data.

Alan Stern

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
linux-usb-***@lists.sourceforge.net
To unsubscribe, use the last form field at:
https://lists.sourceforge.net/lists/listinfo/linux-usb-devel

ilya

2007-11-19 16:05:42 UTC

works like a charm; interesting enough, i've tried this before
but ended up crashing my system so i figured that code
was necessary. so why do you do all this page/block alignment
in the first place if VFS can take care of it?

-- ilya

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
linux-usb-***@lists.sourceforge.net
To unsubscribe, use the last form field at:
https://lists.sourceforge.net/lists/listinfo/linux-usb-devel

Alan Stern

2007-11-19 16:29:07 UTC

Post by Alan Stern
We can fix the problem easily enough. Just take out the code in
do_read() and do_write() that handles partial pages, and let the VFS or
block layer worry about reorganizing the data.

Well, in the first place it isn't really page/block alignment. (That
is, it doesn't involve moving data so that the start address lies on a
page boundary.) It's just reducing the size of a usb_request so that
any additional data will automatically be aligned with no special
effort.

So this was nothing more than a slight optimization. It may not be
worthwhile in the end -- if VFS is going to copy the data into
block-cache pages anyway then the initial alignment won't matter. In
fact, I don't know whether this partial-page stuff actually helps at
all. So I don't mind getting rid of it.

Alan Stern

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
linux-usb-***@lists.sourceforge.net
To unsubscribe, use the last form field at:
https://lists.sourceforge.net/lists/listinfo/linux-usb-devel

David Brownell

2007-11-16 20:37:14 UTC

Post by ilya
this
will be neccessary for wireless usb, among many other
things.

What other things?

He is curious about this idiom of asking questions in the
third person, rather than second person. ;)

Most control requests shouldn't require changes to the
framework.

The NAK stuff seems normal ... if there's no data, nobody
should trigger a STALL even today, see how high speed PING
works.

In terms of changing maxpacket, he's not sure why that would
need to be visible to the gadget driver. As he recalls, a
special value in the endpoint descriptor signifies variable
packet size. And to the extent it's all for bandwidth
management, that can (and hence probably *should*) be done
in the controller driver ... certainly for bulk transfers,
possibly not for periodic transfers (like video/audio,
where less bandwidth requires switching to lower-quality
encodings).

Basically, he'd wait till there's a concrete proposal about
any interface changes ... and would be skeptical about any
which called for significant framework changes for anything
other than ISO data streaming applications. (Of which there
are more or less zero in Linux today...)

- Dave

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
linux-usb-***@lists.sourceforge.net
To unsubscribe, use the last form field at:
https://lists.sourceforge.net/lists/listinfo/linux-usb-devel

David Brownell

2008-01-08 10:29:44 UTC

I know there are at least some parts of this I didn't respond to ...

Post by ilya
wireless usb will be introducing a lot of new control requests,

Are there any that gadget drivers should be aware of? I'd tend
to think that key management should be invisible to them, for
example ... WUSB peripheral controller drivers would interact
directly with the system keystore.

Post by ilya
flow
control [ie: stalling endpoints should be the last resort because in wusb
you can NAK and the host will wait for a special device notification to send
a transfer token instead of constantly polling as in wired usb],

Virtually no drivers other than g_file_storage use stalls for
anything except reporting control transfer errors. So if there's
a new "wireless mass storage" spec, that's what would likely
trigger any relevant API changes.

Stalls have always been a PITA anyway, since hardware behavior
varies so widely. (It's generally nonsensical, treated as an
out-of-band-signal and thus awkward to coordinate with dataflow.)

Post by ilya
association, etc.

In the security/session sense? Again, I'd expect such stuff
to be transparent to the gadget drivers. Experience Has Shown
that applications trying to manage authentication are as a rule
bug-ridden security holes that only accomplish difficulty-of-use
rather than improving security. Maybe it could be made visible
to drivers after the basic infrastructure works smoothly though.

Post by ilya
i think a lot of these things could be handled by the controller driver,
but there are a lot of other stuff that could be standardized and handled
by the "gadget" client side.
what does david brownell think about this?

I'm open to proposals.

In the case of $SUBJECT for example it would make sense to me
for gadget drivers to have a way to learn the curent maxpacket
size for nonperiodic endpoints ... and for periodic ones (notably
ISO), to be able to participate in the maxpacket selection.

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
linux-usb-***@lists.sourceforge.net
To unsubscribe, use the last form field at:
https://lists.sourceforge.net/lists/listinfo/linux-usb-devel

ilya

2008-01-08 18:47:04 UTC

hello david, thank you for your reply. i've been trying to put together
a list of changes to make the Gadget API wireless. i'll post it here
as soon as i am done.

Post by ilya
wireless usb will be introducing a lot of new control requests,

you are right, when i think about it majority of new control requests
relate to the hardware and should be invisible to the gadget driver.
the only request i can think of so far is loopback read/write, which
is described in WUSB spec 1.0 section 4.8.4. basically, it is a way
for a host to test the connection by doing loopback data transfers. the
size of the transfer is determined based on the largest maxpacket of
any device's endpoint. i guess, technically, this can be transparent
to the gadget driver as well.

i don't think there is going to be a new mass storage spec 'till
USB 3. as long as the gadget provides a mechanism
to turn off the stalls, smilar to g_file_storage, everything should
be fine.

Post by ilya
association, etc.

so in order for a new WUSB device to connect with a host it first needs
to be "associated" with it; it is a one-time procedure that says that
in the future this specific device can connect to this specific host.

so far there are two standard mechanism for this [technically three
but i don't know how many controllers will implement near-field association]
i think the numeric association can be kept inside the driver controller;
it is based on the deiffie-hellman key exchange and requires tedious
calculations, but on the other hand it would be nice if it was implemented
inside gadget so in the future any controller driver can support it without
much work.

cable association can definitely be implemented on the gadget side
because it acts as separate wired device. i have already written a generic
cable association gadget. the only problem i am facing is that the
CBAF gadget needs to be aware of the main gadget [g_file_storage,
g_zero, ...] since it needs to provide to the host with some info, such the
device's friendly name, what language it uses, etc.

now, i think this is the most complex part... what if we have a controller
that supports both wired and wireless and maybe can even run two
separate device simultaneously, one on each side. what if when i plugin
my device i want to do cable association but also run the main gadget
wiredly [by advertising two interfaces and so forth, this is very common
among the devices i've seen so far] what if i want to run g_file_storage via
WUSB and simultaneously run g_zero on wired? i think the execution
of multiple gadgets can wait but at a minimum there should be mechanism
of advertising CBAF along with whatever gadget is running on wired and
have one aware of the other.

I'm open to proposals.

soon i am going to post my list of what i think is crucial to make the
Gadget API wireless and what would be nice to have and we can go
from there.

again, i appreciate your participation.

-- ilya

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
linux-usb-***@lists.sourceforge.net
To unsubscribe, use the last form field at:
https://lists.sourceforge.net/lists/listinfo/linux-usb-devel

David Brownell

2008-01-08 19:44:25 UTC

Post by ilya
association, etc.

What I'd actually like is to have this packaged so that it can
work with wired links too. The issue isn't calculation ... but
how to securely store the keys. Remember that the security of
the system depends on security of those keys. Imagine that they
are managed in separate crypto hardware module, and they physically
can't leave that module. Keys are then completely inaccessible to
any gadget driver. That will give the right model for software.

That applies equally on the host side, of course.

Post by ilya
cable association can definitely be implemented on the gadget side
because it acts as separate wired device. i have already written a generic
cable association gadget. the only problem i am facing is that the
CBAF gadget needs to be aware of the main gadget [g_file_storage,
g_zero, ...] since it needs to provide to the host with some info, such the
device's friendly name, what language it uses, etc.

I've not looked at this stuff in a long time, but I'd expect that
it should be possible to let the standard EP0 interfaces to the
gadget driver handle all that stuff ... and just punt unrecognized
control requests (that happen to be for association management) to
the relevant module.

Post by ilya
now, i think this is the most complex part... what if we have a controller
that supports both wired and wireless and maybe can even run two
separate device simultaneously, one on each side.

Presenting two separate gadgets would be a bit of a stretch. :)

Post by ilya
what if when i plugin
my device i want to do cable association but also run the main gadget
wiredly [by advertising two interfaces and so forth, this is very common
among the devices i've seen so far] what if i want to run g_file_storage via
WUSB and simultaneously run g_zero on wired? i think the execution
of multiple gadgets can wait but at a minimum there should be mechanism
of advertising CBAF along with whatever gadget is running on wired and
have one aware of the other.

To save myself the effort of re-reading specs (and other folk on this
list from reading them the first time) ... is the ability to support
associations exposed in the config descriptors?

If so, then the gadget drivers would need updating to support this
scheme. And then they could reasonably just delegate unrecognized
setup calls to a library module which manages associations.

I'm open to proposals.

soon i am going to post my list of what i think is crucial to make the
Gadget API wireless and what would be nice to have and we can go
from there.

OK, looking forward to see that proposal.

- Dave

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
linux-usb-***@lists.sourceforge.net
To unsubscribe, use the last form field at:
https://lists.sourceforge.net/lists/listinfo/linux-usb-devel

vichy

2008-01-15 07:37:21 UTC

Dear all:
Dear USB friends:
I use FPGA to simulate a device with different subclass and protocols
defined in usb2.0.
But I have no idea where can I get tools running on Linux to send different
standard requests to test my device.
I know maybe I can write another module based on usbcore.ko, which passes
urbs with different pipes to host directly, but I just want to know whether
there is already one or some kind person can share his with me.
Appreciate your help,
vichy

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
linux-usb-***@lists.sourceforge.net
To unsubscribe, use the last form field at:
https://lists.sourceforge.net/lists/listinfo/linux-usb-devel

David Brownell

2008-01-15 07:45:41 UTC

Post by vichy
I use FPGA to simulate a device with different subclass and protocols
defined in usb2.0.
But I have no idea where can I get tools running on Linux to send different
standard requests to test my device.

Start with http://www.linux-usb.org/usbtest ...

Post by vichy
I know maybe I can write another module based on usbcore.ko, which passes
urbs with different pipes to host directly, but I just want to know whether
there is already one or some kind person can share his with me.
Appreciate your help,
vichy

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
linux-usb-***@lists.sourceforge.net
To unsubscribe, use the last form field at:
https://lists.sourceforge.net/lists/listinfo/linux-usb-devel

Felipe Balbi

2007-11-14 18:41:50 UTC

Hi,

On Wed, 14 Nov 2007 12:02:31 -0500 (EST), Alan Stern

Then where would the maxpacket field in the usb_ep structure get
initialized in the first place? A module option would be possible.

wouldn't it shit up usb specs ?
maxpacket field should be static for a vendorid:productid pair.

Maybe declaring it the biggest possible packet and
treating/creating/sending
usb packets with different sizes internally would look better for usb-if ?

Post by ilya
this
will be neccessary for wireless usb, among many other
things.

What other things?

Post by ilya
i am not really familiar with writing to disk from
kernel space, so any suggestions would be appreciated.

Changing the high-speed maxpacket size wouldn't be hard. It would
if (gadget_is_dualspeed(gadget)) {
in the fsg_bind() routine. It wouldn't affect writing to disk at all.
Alan Stern
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
https://lists.sourceforge.net/lists/listinfo/linux-usb-devel

--
Best Regards,

Felipe Balbi
http://felipebalbi.com
***@felipebalbi.com

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
linux-usb-***@lists.sourceforge.net
To unsubscribe, use the last form field at:
https://lists.sourceforge.net/lists/listinfo/linux-usb-devel

Alan Stern

2007-11-15 14:52:53 UTC

Post by Felipe Balbi