infrastructure/fedora-infrastructure

Fork 0

Dist -git in staging does not support shallow clients #8384

New issue

Closed

opened 2019-11-14 12:38:00 +00:00 by cverna · 12 comments

cverna commented

2019-11-14 12:38:00 +00:00

While testing OSBS in staging I have got the following error. I suspect this is related to repoSpanner but I am not 100% sure that's why I create the ticket here.

git clone -b f29 --single-branch --depth 1 https://src.stg.fedoraproject.org/container/origin-base.git
Cloning into 'origin-base'...
fatal: Server does not support shallow clients
fatal: the remote end hung up unexpectedly

koji tasks : https://koji.stg.fedoraproject.org/koji/taskinfo?taskID=90169916

While testing OSBS in staging I have got the following error. I suspect this is related to repoSpanner but I am not 100% sure that's why I create the ticket here. ``` git clone -b f29 --single-branch --depth 1 https://src.stg.fedoraproject.org/container/origin-base.git Cloning into 'origin-base'... fatal: Server does not support shallow clients fatal: the remote end hung up unexpectedly ``` koji tasks : https://koji.stg.fedoraproject.org/koji/taskinfo?taskID=90169916

cverna commented

2019-11-14 12:38:34 +00:00

Author

@bowlofeggs do you know if that would be related to repoSpanner ?

cc: @jcajka

@bowlofeggs do you know if that would be related to repoSpanner ? cc: @jcajka

cverna commented

2019-11-14 12:38:54 +00:00

Author

Metadata Update from @cverna:

Issue tagged with: OSBS, repoSpanner

**Metadata Update from @cverna**: - Issue tagged with: OSBS, repoSpanner

bowlofeggs commented

2019-11-14 15:27:59 +00:00

On Thu, 2019-11-14 at 12:38 +0000, Clement Verna wrote:

@bowlofeggs do you know if that would be related to repoSpanner ?=20

It could be, but I don't know for sure.

On Thu, 2019-11-14 at 12:38 +0000, Clement Verna wrote: > @bowlofeggs do you know if that would be related to repoSpanner ?=20 It could be, but I don't know for sure.

kevin commented

2019-11-14 18:54:54 +00:00

@bowlofeggs could you test in your test env and confirm it does the same thing?

Is this functionality we need? or can osbs be told to not do that?

@bowlofeggs could you test in your test env and confirm it does the same thing? Is this functionality we need? or can osbs be told to not do that?

kevin commented

2019-11-14 18:54:54 +00:00

Metadata Update from @kevin:

Issue priority set to: Waiting on Assignee (was: Needs Review)

**Metadata Update from @kevin**: - Issue priority set to: Waiting on Assignee (was: Needs Review)

cverna commented

2019-11-14 19:59:52 +00:00

Author

@bowlofeggs could you test in your test env and confirm it does the same thing?
Is this functionality we need? or can osbs be told to not do that?

This is hardcoded in OSBS (see logs below), I am pretty sure they are doing this for performance to make the build faster. To continue testing in stg I can try to make a patch and just use a plain git clone but ideally I would not want to carry that patch forever 😄

2019-11-14 12:38:43,950 - osbs.utils - DEBUG - url = 'https://src.stg.fedoraproject.org/container/origin-base.git', dir = '/tmp/tmpb9veecci/origin-base', commit = '7c26a383f19c365ca4c29483ee5ca7773a2eb344'
2019-11-14 12:38:43,950 - osbs.utils - DEBUG - cloning '['git', 'clone', '-b', 'f29', '--single-branch', '--depth', '1', 'https://src.stg.fedoraproject.org/container/origin-base.git', '/tmp/tmpb9veecci/origin-base']'

> @bowlofeggs could you test in your test env and confirm it does the same thing? > Is this functionality we need? or can osbs be told to not do that? This is hardcoded in OSBS (see logs below), I am pretty sure they are doing this for performance to make the build faster. To continue testing in stg I can try to make a patch and just use a plain `git clone` but ideally I would not want to carry that patch forever :smile: ``` 2019-11-14 12:38:43,950 - osbs.utils - DEBUG - url = 'https://src.stg.fedoraproject.org/container/origin-base.git', dir = '/tmp/tmpb9veecci/origin-base', commit = '7c26a383f19c365ca4c29483ee5ca7773a2eb344' 2019-11-14 12:38:43,950 - osbs.utils - DEBUG - cloning '['git', 'clone', '-b', 'f29', '--single-branch', '--depth', '1', 'https://src.stg.fedoraproject.org/container/origin-base.git', '/tmp/tmpb9veecci/origin-base']' ```

jlanda commented

2019-11-15 12:57:59 +00:00

Some info..

curl -i --output - https://src.stg.fedoraproject.org/container/origin-base.git/info/refs?service=git-upload-pack:



        HTTP/1.1 200 OK
        Date: Fri, 15 Nov 2019 12:53:02 GMT
        Server: Apache
        Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
        X-Frame-Options: SAMEORIGIN
        X-Xss-Protection: 1; mode=block
        X-Content-Type-Options: nosniff
        Referrer-Policy: same-origin
        x-xss-protection: 1; mode=block
        Content-Security-Policy: default-src 'self';script-src 'self' 'nonce-N930qHWfpNNZMkgUpWJFwWf2T' https://apps.fedoraproject.org; style-src 'self' 'nonce-N930qHWfpNNZMkgUpWJFwWf2T'; object-src 'none';base-uri 'self';img-src 'self' https:;
        x-content-type-options: nosniff
        strict-transport-security: max-age=31536000; includeSubDomains; preload
        cache-control: no-cache
        x-frame-options: DENY
        referrer-policy: no-referrer
        x-repospanner-nodename: fedora01
        AppTime: D=159441
        X-Fedora-AppServer: pkgs01.stg.phx2.fedoraproject.org
        Content-Type: application/x-git-upload-pack-advertisement
        Set-Cookie: disgit_pagure=eyJfcGVybWFuZW50Ijp0cnVlLCJjc3JmIjp7IiBiIjoiWWpBeU1qbGhaRFZpTkRWbE1tVTRaV000WW1Wa09UQTBaakUxTnpBM09UQXhObUpsTW1ZMlpRPT0ifX0.ELAxLg.dm6bUqYES9ziDeM2b2R5vbgWAFU; Expires=Mon, 16-Dec-2019 12:53:02 GMT; Secure; HttpOnly; Path=/
        X-Fedora-ProxyServer: proxy01.stg.phx2.fedoraproject.org
        X-Fedora-RequestID: Xc6frut8AWer-zc594cwEAAACQA
        Transfer-Encoding: chunked
         
        001e# service=git-upload-pack
        00000139e7a5e0104efbf6d231196a00c0ec4bb15809321a refs/heads/f28delete-refs no-thin no-done side-band side-band-64k report-status allow-tip-sha1-in-want multi_ack multi_ack_detailed allow-reachable-sha1-in-want agent=repoSpanner/0.5+11.c6a026e428baaa9b34c8fd54a6f1c0a6f2ae062e.el7.infra symref=HEAD:refs/heads/master
        003b3a2f054acfdd575975cbb5915a85acd16ecf5594 refs/heads/f29003eac337d19f76b1aab7ad6c4a4de3f1f29a8d60c66 refs/heads/master0049c3e6b35d469274f1f22a8d697de7032085868ab9 refs/heads/test_fedora_image003d211d3a56d1e0221f65c5ef689e41eabd14fd8765 refs/pull/1/head003b211d3a56d1e0221f65c5ef689e41eabd14fd8765 refs/heads/f270031ac337d19f76b1aab7ad6c4a4de3f1f29a8d60c66 HEAD0000

there is no swallow thin and so when repospanner responds.

Now the same thing without repospanner:
curl -i --output - https://pagure.io/pagure.git/info/refs?service=git-upload-pack:

HTTP/1.1 200 OK
Date: Fri, 15 Nov 2019 12:56:17 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5
expires: Fri, 01 Jan 1980 00:00:00 GMT
pragma: no-cache
cache-control: no-cache, max-age=0, must-revalidate
Content-Security-Policy: default-src 'self' https:; script-src 'self' 'unsafe-eval' 'unsafe-inline' https://apps.fedoraproject.org; style-src 'self' 'unsafe-inline' https://apps.fedoraproject.org
Set-Cookie: pagure=eyJfcGVybWFuZW50Ijp0cnVlLCJjc3JmIjp7IiBiIjoiTlRoaU1HTm1OVEkzWVdRM1kyUTJNemd6TnpOaVptRXdNMkkwTVdRNE5qUmpOakk1TTJSbE1RPT0ifX0.ELAx8Q.umcwoqNFV-oHtWbjbpB5yK1_q-g; Expires=Mon, 16-Dec-2019 12:56:17 GMT; Secure; HttpOnly; Path=/
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
X-Frame-Options: ALLOW-FROM https://pagure.io/
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Transfer-Encoding: chunked
Content-Type: application/x-git-upload-pack-advertisement

001e# service=git-upload-pack
000000b5eb7ca5eb60f12ff17e580e3777db492c2a64815f HEADmulti_ack thin-pack side-band side-band-64k ofs-delta shallow no-progress include-tag multi_ack_detailed no-done agent=git/1.8.3.1

git on pagure.io responds with shallow thin-pack etc on it's capabilities, repospanner does not :)

Some info.. `curl -i --output - https://src.stg.fedoraproject.org/container/origin-base.git/info/refs?service=git-upload-pack`: ``` HTTP/1.1 200 OK Date: Fri, 15 Nov 2019 12:53:02 GMT Server: Apache Strict-Transport-Security: max-age=31536000; includeSubDomains; preload X-Frame-Options: SAMEORIGIN X-Xss-Protection: 1; mode=block X-Content-Type-Options: nosniff Referrer-Policy: same-origin x-xss-protection: 1; mode=block Content-Security-Policy: default-src 'self';script-src 'self' 'nonce-N930qHWfpNNZMkgUpWJFwWf2T' https://apps.fedoraproject.org; style-src 'self' 'nonce-N930qHWfpNNZMkgUpWJFwWf2T'; object-src 'none';base-uri 'self';img-src 'self' https:; x-content-type-options: nosniff strict-transport-security: max-age=31536000; includeSubDomains; preload cache-control: no-cache x-frame-options: DENY referrer-policy: no-referrer x-repospanner-nodename: fedora01 AppTime: D=159441 X-Fedora-AppServer: pkgs01.stg.phx2.fedoraproject.org Content-Type: application/x-git-upload-pack-advertisement Set-Cookie: disgit_pagure=eyJfcGVybWFuZW50Ijp0cnVlLCJjc3JmIjp7IiBiIjoiWWpBeU1qbGhaRFZpTkRWbE1tVTRaV000WW1Wa09UQTBaakUxTnpBM09UQXhObUpsTW1ZMlpRPT0ifX0.ELAxLg.dm6bUqYES9ziDeM2b2R5vbgWAFU; Expires=Mon, 16-Dec-2019 12:53:02 GMT; Secure; HttpOnly; Path=/ X-Fedora-ProxyServer: proxy01.stg.phx2.fedoraproject.org X-Fedora-RequestID: Xc6frut8AWer-zc594cwEAAACQA Transfer-Encoding: chunked 001e# service=git-upload-pack 00000139e7a5e0104efbf6d231196a00c0ec4bb15809321a refs/heads/f28delete-refs no-thin no-done side-band side-band-64k report-status allow-tip-sha1-in-want multi_ack multi_ack_detailed allow-reachable-sha1-in-want agent=repoSpanner/0.5+11.c6a026e428baaa9b34c8fd54a6f1c0a6f2ae062e.el7.infra symref=HEAD:refs/heads/master 003b3a2f054acfdd575975cbb5915a85acd16ecf5594 refs/heads/f29003eac337d19f76b1aab7ad6c4a4de3f1f29a8d60c66 refs/heads/master0049c3e6b35d469274f1f22a8d697de7032085868ab9 refs/heads/test_fedora_image003d211d3a56d1e0221f65c5ef689e41eabd14fd8765 refs/pull/1/head003b211d3a56d1e0221f65c5ef689e41eabd14fd8765 refs/heads/f270031ac337d19f76b1aab7ad6c4a4de3f1f29a8d60c66 HEAD0000 ``` there is no swallow thin and so when repospanner responds. Now the same thing without repospanner: `curl -i --output - https://pagure.io/pagure.git/info/refs?service=git-upload-pack`: ``` HTTP/1.1 200 OK Date: Fri, 15 Nov 2019 12:56:17 GMT Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5 expires: Fri, 01 Jan 1980 00:00:00 GMT pragma: no-cache cache-control: no-cache, max-age=0, must-revalidate Content-Security-Policy: default-src 'self' https:; script-src 'self' 'unsafe-eval' 'unsafe-inline' https://apps.fedoraproject.org; style-src 'self' 'unsafe-inline' https://apps.fedoraproject.org Set-Cookie: pagure=eyJfcGVybWFuZW50Ijp0cnVlLCJjc3JmIjp7IiBiIjoiTlRoaU1HTm1OVEkzWVdRM1kyUTJNemd6TnpOaVptRXdNMkkwTVdRNE5qUmpOakk1TTJSbE1RPT0ifX0.ELAx8Q.umcwoqNFV-oHtWbjbpB5yK1_q-g; Expires=Mon, 16-Dec-2019 12:56:17 GMT; Secure; HttpOnly; Path=/ X-Xss-Protection: 1; mode=block X-Content-Type-Options: nosniff Referrer-Policy: same-origin X-Frame-Options: ALLOW-FROM https://pagure.io/ Strict-Transport-Security: max-age=31536000; includeSubDomains; preload Transfer-Encoding: chunked Content-Type: application/x-git-upload-pack-advertisement 001e# service=git-upload-pack 000000b5eb7ca5eb60f12ff17e580e3777db492c2a64815f HEADmulti_ack thin-pack side-band side-band-64k ofs-delta shallow no-progress include-tag multi_ack_detailed no-done agent=git/1.8.3.1 ``` git on pagure.io responds with shallow thin-pack etc on it's capabilities, repospanner does not :)

jlanda commented

2019-11-15 13:04:38 +00:00

eeb960cb61/server/service/gitutils.go (L171-L183) <-- this sets the extensions on repospanner . Will just adding shallow thin and removing no-thin work? no idea :D

https://github.com/repoSpanner/repoSpanner/blob/eeb960cb612a97201d88a7bd82c954fb9cf7ef44/server/service/gitutils.go#L171-L183 <-- this sets the extensions on repospanner . Will just adding `shallow thin` and removing `no-thin` work? no idea :D

kevin commented

2019-11-19 18:12:09 +00:00

Yeah, not sure what the next action is here... how about a upstream repospanner ticket/issue to support shallow?

@cverna can you file that and close this?

Yeah, not sure what the next action is here... how about a upstream repospanner ticket/issue to support shallow? @cverna can you file that and close this?

cverna commented

2019-11-19 18:21:53 +00:00

Author

Done https://github.com/repoSpanner/repoSpanner/issues/104

cverna commented

2019-11-19 18:21:54 +00:00

Author

Metadata Update from @cverna:

Issue close_status updated to: Fixed
Issue status updated to: Closed (was: Open)

**Metadata Update from @cverna**: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)

bowlofeggs commented

2019-11-19 19:41:47 +00:00

@cverna @kevin Yeah I get this when I try a shallow clone from my local repospanner:

fatal: Server does not support shallow clients
fatal: the remote end hung up unexpectedly

@cverna @kevin Yeah I get this when I try a shallow clone from my local repospanner: ``` fatal: Server does not support shallow clients fatal: the remote end hung up unexpectedly ```