Address remaining issues related to local serving of federation logos:
it should be not possible to start a new logos downloading process, while one is still in progress (NOTE: update documentation to reflect this).
logos should be available as soon as are downloaded, not only after a full download of all; i.e. move from stage dir to final dir as soon as logo is downloaded; then add pointer file w/ extension
from time to time clean logos of federations which are not in production anymore. Note that this is important especially in case of switch AAI, which is… rotating their federation id very often, it is including a timestamp. (work moved to a separate ticket)
Clean logos of federations which are still in production. Remove all logo that were not fetched in iteration.
Experiment with connection vs read timeouts. Perhaps we can have better efficiency with separate controls (longer connect vs shorter read)?
loading of logos which were not prefetched (i.e. not coming from the federation) was broken in some cases
there was serious congestion on the executor when thousands of logo download tasks were plugged into it, essentially postponing all other tasks until almost all logos were downloaded.
I’ve fixed all those issues, will open CRs for them.
Address remaining issues related to local serving of federation logos:
it should be not possible to start a new logos downloading process, while one is still in progress (NOTE: update documentation to reflect this).
logos should be available as soon as are downloaded, not only after a full download of all; i.e. move from stage dir to final dir as soon as logo is downloaded; then add pointer file w/ extension
from time to time clean logos of federations which are not in production anymore. Note that this is important especially in case of switch AAI, which is… rotating their federation id very often, it is including a timestamp. (work moved to a separate ticket)
Clean logos of federations which are still in production. Remove all logo that were not fetched in iteration.
Experiment with connection vs read timeouts. Perhaps we can have better efficiency with separate controls (longer connect vs shorter read)?
Investigate reasons behind unexpected failures. Unavailable server, 404 and SSL problems are possible and expected. But why https://shibboleth.coventrycollege.ac.uk/logo-medium.png is not downloaded?