Develop: Zap stays in Scanner

[wurstbrot]

I am using docker-compose.yml with a self build engine and zap. Both are current develop-branch without modifications.
I scan the juice shop via swagger:
[ { "name": "ZAP Scan 10.10.11.104", "location": "http://10.10.11.104:3000/", "attributes": { "ZAP_BASE_URL": "http://10.10.11.104:3000/", "ZAP_SPIDER_MAX_DEPTH": 1 } } ]

I am not sure which warning/error in the following log is the cause for zap to stay Scanner after finishing "Run OWASP Zap Spider" in the engine.

I see the following in the zap-container-logs:

214907 [ZAP-ProxyThread-70] INFO org.zaproxy.zap.extension.httpsessions.HttpSessionsSite  - Setting new active session for site '10.10.11.104:3000': HttpSession [name=secureCodeBoxSession, active=false, tokenValues='']
214911 [ZAP-ProxyThread-71] INFO org.zaproxy.zap.extension.httpsessions.HttpSessionsSite  - Setting new active session for site '10.10.11.104:3000': HttpSession [name=secureCodeBoxSession, active=true, tokenValues='']
2018-11-16 12:50:00.561  INFO 118 --- [pool-1-thread-4] i.s.zap.service.zap.ZapService           : Recalling 0 requests to zap.
2018-11-16 12:50:00.561  INFO 118 --- [pool-1-thread-4] i.s.zap.service.zap.ZapService           : Starting scanner for targetUrl 'http://10.10.11.104:3000/main.js' and userId -1.
2018-11-16 12:50:00.594  INFO 118 --- [pool-1-thread-4] i.s.zap.service.zap.ZapService           : No custom ZAP replacer rule defined yet.
214949 [ZAP-ProxyThread-78] WARN org.zaproxy.zap.extension.api.API  - Bad request to API endpoint [/xml/ascan/action/scan/] from [127.0.0.1]:
URL Not Found in the Scan Tree (url_not_found)
	at org.zaproxy.zap.extension.ascan.ActiveScanAPI.scanURL(ActiveScanAPI.java:779)
	at org.zaproxy.zap.extension.ascan.ActiveScanAPI.handleApiAction(ActiveScanAPI.java:293)
	at org.zaproxy.zap.extension.api.API.handleApiRequest(API.java:431)
	at org.parosproxy.paros.core.proxy.ProxyThread.processHttp(ProxyThread.java:456)
	at org.parosproxy.paros.core.proxy.ProxyThread.run(ProxyThread.java:317)
	at java.lang.Thread.run(Thread.java:748)
2018-11-16 12:50:00.604 ERROR 118 --- [pool-1-thread-4] i.s.zap.jobs.definition.EngineWorkerJob  : Job execution error!

org.zaproxy.clientapi.core.ClientApiException: URL Not Found in the Scan Tree
	at org.zaproxy.clientapi.core.ApiResponseFactory.getResponse(ApiResponseFactory.java:50) ~[zap-clientapi-1.6.0.jar!/:1.6.0]
	at org.zaproxy.clientapi.core.ClientApi.callApi(ClientApi.java:332) ~[zap-clientapi-1.6.0.jar!/:1.6.0]
	at org.zaproxy.clientapi.gen.Ascan.scan(Ascan.java:278) ~[zap-clientapi-1.6.0.jar!/:1.6.0]
	at org.zaproxy.clientapi.gen.Ascan.scan(Ascan.java:236) ~[zap-clientapi-1.6.0.jar!/:1.6.0]
	at io.securecodebox.zap.service.zap.ZapService.startScannerAsUser(ZapService.java:260) ~[classes!/:na]
	at io.securecodebox.zap.jobs.definition.EngineWorkerJob.executeScanner(EngineWorkerJob.java:239) ~[classes!/:na]
	at io.securecodebox.zap.jobs.definition.EngineWorkerJob.performScannerTask(EngineWorkerJob.java:179) ~[classes!/:na]
	at io.securecodebox.zap.jobs.definition.EngineWorkerJob.execute(EngineWorkerJob.java:108) ~[classes!/:na]
	at de.otto.edison.jobs.service.JobService$1.execute(JobService.java:252) [edison-jobs-0.82.2.jar!/:na]
	at de.otto.edison.jobs.service.JobRunner.executeAndRetry(JobRunner.java:61) [edison-jobs-0.82.2.jar!/:na]
	at de.otto.edison.jobs.service.JobRunner.start(JobRunner.java:50) [edison-jobs-0.82.2.jar!/:na]
	at de.otto.edison.jobs.service.JobService.lambda$startAsync$7(JobService.java:216) [edison-jobs-0.82.2.jar!/:na]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_131]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_131]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]

2018-11-16 12:50:00.728 ERROR 118 --- [pool-1-thread-4] d.o.e.jobs.eventbus.LogJobEventListener  : 'Fatal error in job engine/worker/owasp/zap (64287835-c2f1-42bd-aee7-e602b53f5034)
org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://engine:8080/box/jobs/ff1825e9-e99d-11e8-a098-0242ac120006/failure": http://engine:8080/box/jobs/ff1825e9-e99d-11e8-a098-0242ac120006/failure; nested exception is java.io.FileNotFoundException: http://engine:8080/box/jobs/ff1825e9-e99d-11e8-a098-0242ac120006/failure
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:674)
	at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:621)
	at org.springframework.web.client.RestTemplate.postForEntity(RestTemplate.java:415)
	at io.securecodebox.zap.service.engine.EngineTaskApiClient.reportFailure(EngineTaskApiClient.java:145)
	at io.securecodebox.zap.service.engine.ZapTaskService.reportFailure(ZapTaskService.java:66)
	at io.securecodebox.zap.jobs.definition.EngineWorkerJob.execute(EngineWorkerJob.java:111)
	at de.otto.edison.jobs.service.JobService$1.execute(JobService.java:252)
	at de.otto.edison.jobs.service.JobRunner.executeAndRetry(JobRunner.java:61)
	at de.otto.edison.jobs.service.JobRunner.start(JobRunner.java:50)
	at de.otto.edison.jobs.service.JobService.lambda$startAsync$7(JobService.java:216)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: http://engine:8080/box/jobs/ff1825e9-e99d-11e8-a098-0242ac120006/failure
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1872)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
	at org.springframework.http.client.SimpleClientHttpResponse.getBody(SimpleClientHttpResponse.java:85)
	at org.springframework.http.client.BufferingClientHttpResponseWrapper.getBody(BufferingClientHttpResponseWrapper.java:69)
	at io.securecodebox.zap.service.engine.LoggingRequestInterceptor.traceResponse(LoggingRequestInterceptor.java:58)
	at io.securecodebox.zap.service.engine.LoggingRequestInterceptor.intercept(LoggingRequestInterceptor.java:44)
	at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:88)
	at org.springframework.http.client.InterceptingClientHttpRequest.executeInternal(InterceptingClientHttpRequest.java:72)
	at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
	at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:660)
	... 16 more
': '64287835-c2f1-42bd-aee7-e602b53f5034'
2018-11-16 12:50:00.730 ERROR 118 --- [pool-1-thread-4] de.otto.edison.jobs.service.JobRunner    : Fatal error in job engine/worker/owasp/zap (64287835-c2f1-42bd-aee7-e602b53f5034)

org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://engine:8080/box/jobs/ff1825e9-e99d-11e8-a098-0242ac120006/failure": http://engine:8080/box/jobs/ff1825e9-e99d-11e8-a098-0242ac120006/failure; nested exception is java.io.FileNotFoundException: http://engine:8080/box/jobs/ff1825e9-e99d-11e8-a098-0242ac120006/failure
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:674) ~[spring-web-4.3.15.RELEASE.jar!/:4.3.15.RELEASE]
	at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:621) ~[spring-web-4.3.15.RELEASE.jar!/:4.3.15.RELEASE]
	at org.springframework.web.client.RestTemplate.postForEntity(RestTemplate.java:415) ~[spring-web-4.3.15.RELEASE.jar!/:4.3.15.RELEASE]
	at io.securecodebox.zap.service.engine.EngineTaskApiClient.reportFailure(EngineTaskApiClient.java:145) ~[classes!/:na]
	at io.securecodebox.zap.service.engine.ZapTaskService.reportFailure(ZapTaskService.java:66) ~[classes!/:na]
	at io.securecodebox.zap.jobs.definition.EngineWorkerJob.execute(EngineWorkerJob.java:111) ~[classes!/:na]
	at de.otto.edison.jobs.service.JobService$1.execute(JobService.java:252) ~[edison-jobs-0.82.2.jar!/:na]
	at de.otto.edison.jobs.service.JobRunner.executeAndRetry(JobRunner.java:61) [edison-jobs-0.82.2.jar!/:na]
	at de.otto.edison.jobs.service.JobRunner.start(JobRunner.java:50) [edison-jobs-0.82.2.jar!/:na]
	at de.otto.edison.jobs.service.JobService.lambda$startAsync$7(JobService.java:216) [edison-jobs-0.82.2.jar!/:na]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_131]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_131]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.io.FileNotFoundException: http://engine:8080/box/jobs/ff1825e9-e99d-11e8-a098-0242ac120006/failure
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1872) ~[na:1.8.0_131]
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474) ~[na:1.8.0_131]
	at org.springframework.http.client.SimpleClientHttpResponse.getBody(SimpleClientHttpResponse.java:85) ~[spring-web-4.3.15.RELEASE.jar!/:4.3.15.RELEASE]
	at org.springframework.http.client.BufferingClientHttpResponseWrapper.getBody(BufferingClientHttpResponseWrapper.java:69) ~[spring-web-4.3.15.RELEASE.jar!/:4.3.15.RELEASE]
	at io.securecodebox.zap.service.engine.LoggingRequestInterceptor.traceResponse(LoggingRequestInterceptor.java:58) ~[classes!/:na]
	at io.securecodebox.zap.service.engine.LoggingRequestInterceptor.intercept(LoggingRequestInterceptor.java:44) ~[classes!/:na]
	at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:88) ~[spring-web-4.3.15.RELEASE.jar!/:4.3.15.RELEASE]
	at org.springframework.http.client.InterceptingClientHttpRequest.executeInternal(InterceptingClientHttpRequest.java:72) ~[spring-web-4.3.15.RELEASE.jar!/:4.3.15.RELEASE]
	at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-4.3.15.RELEASE.jar!/:4.3.15.RELEASE]
	at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web-4.3.15.RELEASE.jar!/:4.3.15.RELEASE]
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:660) ~[spring-web-4.3.15.RELEASE.jar!/:4.3.15.RELEASE]

[rfelber]

Hi Timo,

you LOG file indicates that your ZAP Spider returns no URLs, in this case the ZAP Scanner complains that its not able to scan...

** 2018-11-16 12:50:00.561 INFO 118 --- [pool-1-thread-4] i.s.zap.service.zap.ZapService : Recalling 0 requests to zap.**

214949 [ZAP-ProxyThread-78] WARN org.zaproxy.zap.extension.api.API - Bad request to API endpoint [/xml/ascan/action/scan/] from [127.0.0.1]:
URL Not Found in the Scan Tree (url_not_found)

Have you tried to scan your service (http://10.10.11.104:3000/) local with ZAP first, to test your configuration you would like to automate? Maybe the ZAP_SPIDER_MAX_DEPTH must be >= 1?

KR
Robert

[wurstbrot]

Hi Robert,

I tried it now with:

• ZAP_SPIDER_MAX_DEPTH=2
• Without ZAP_SPIDER_MAX_DEPTH
• With bodgeit with ZAP_SPIDER_MAX_DEPTH=1 according to documentation (https://github.com/secureCodeBox/secureCodeBox/blob/master/docs/user-guide/usage-examples/zap-bodgeit-example.md)

In all cases, I get the same error.
In engine I get:
engine_1 | 2018-11-16 14:19:02.640 INFO 1 --- [nio-8080-exec-6] s.s.l.TransformFindingsToTargetsListener : Created Targets out of Findings: [Target{name='null', location='http://bodgeit:8080/bodgeit/js/util.js', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/js/util.js, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}, {name=b_id, value=2}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=b_id=2; JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=323, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/search.jsp', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/search.jsp, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=315, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/basket.jsp', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/basket.jsp, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=315, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/product.jsp?typeid=5', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/product.jsp?typeid=5, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}, {name=b_id, value=2}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=b_id=2; JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[{name=typeid, value=5}], postData={mimeType=, params=[], text=}, headersSize=333, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/home.jsp', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/home.jsp, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=313, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/.svn/entries', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/.svn/entries, httpVersion=HTTP/1.1, cookies=[], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Host, value=bodgeit:8080}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=225, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/style.css', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/style.css, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}, {name=b_id, value=2}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=b_id=2; JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=322, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/login.jsp', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/login.jsp, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=314, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/admin.jsp', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/admin.jsp, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}, {name=b_id, value=2}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=b_id=2; JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=322, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/contact.jsp', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/contact.jsp, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=316, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/.svn/wc.db', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/.svn/wc.db, httpVersion=HTTP/1.1, cookies=[], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Host, value=bodgeit:8080}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=223, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/product.jsp?prodid=2', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/product.jsp?prodid=2, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}, {name=b_id, value=2}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=b_id=2; JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[{name=prodid, value=2}], postData={mimeType=, params=[], text=}, headersSize=333, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/.git/index', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/.git/index, httpVersion=HTTP/1.1, cookies=[], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Host, value=bodgeit:8080}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=223, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/, httpVersion=HTTP/1.1, cookies=[], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Host, value=bodgeit:8080}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=213, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/about.jsp', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/about.jsp, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=314, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}]

I updated the title, as it stays in scanner, saying the task spider is complete.

Cheers,
Timo

[wurstbrot]

I guess due to the error message, that the targetUrl is not in the siteTree and the source of the error is here:
https://github.com/igorhvr/zaproxy/blob/master/src/org/zaproxy/zap/extension/ascan/ActiveScanAPI.java#L98

[J12934]

Hi Timo
Can you please double check if your engine version really is from the current develop branch.
The log from the engine looks to me as it is not. We have recently changed the format zap spider findings get transformed into and this appears to still be the old format.

[wurstbrot]

First test: With the current development version and a local build it works. More tests tomorrow.

[wurstbrot]

Using latest securecodebox/engine:develop and securecodebox/zap:develop (using docker-compose pull) results in an error on the engine.

Used versions:
securecodebox/engine develop 0af6e826d7c8 2 days ago 172MB
securecodebox/zap develop 019df2720f2d 9 days ago 559MB

Logs:

2018-11-22 07:49:31.948 ERROR 9 --- [nio-8080-exec-8] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception

org.springframework.security.web.firewall.RequestRejectedException: The request was rejected because the URL was not normalized.
	at org.springframework.security.web.firewall.StrictHttpFirewall.getFirewalledRequest(StrictHttpFirewall.java:248) ~[spring-security-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:193) ~[spring-security-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) ~[spring-security-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106) ~[spring-boot-actuator-1.5.13.RELEASE.jar!/:1.5.13.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]

With the self build of the current master of github it works. By using develop branch of engine, it is not working. Maybe I am forced to use the new api (which is hard without documentation)?

[wurstbrot]

The last problem doesn't has to be a problem with zap.
The engine shows that error continuously and zap does it's job.
The test happend with the images from this morning (engine+zap from develop):
$ docker images | grep securecodebox
securecodebox/engine develop 5191b1e2f556 20 hours ago 172MB
securecodebox/arachni develop c2c858d4fe64 20 hours ago 1.88GB
securecodebox/engine 0af6e826d7c8 3 days ago 172MB
securecodebox/engine f4547a285c98 7 days ago 165MB
securecodebox/zap develop 019df2720f2d 10 days ago 559MB
securecodebox/arachni eed1c249881c 10 days ago 1.88GB
securecodebox/amass oss c42e42ec9c17 2 months ago 31.5MB
securecodebox/nikto v0.9.0 22f0967523fa 2 months ago 285MB
securecodebox/arachni v0.9.0 80649130b684 2 months ago 1.88GB
securecodebox/sslyze v0.9.0 0794f118d43c 2 months ago 1.05GB
securecodebox/nmap v0.9.0 6799a25e1cca 2 months ago 91.5MB
securecodebox/bodgeit latest d7eb6d6890bd 9 months ago 127MB

[J12934]

Using latest securecodebox/engine:develop and securecodebox/zap:develop (using docker-compose pull) results in an error on the engine.

Used versions:
securecodebox/engine develop 0af6e826d7c8 2 days ago 172MB
securecodebox/zap develop 019df2720f2d 9 days ago 559MB

Logs:

2018-11-22 07:49:31.948 ERROR 9 --- [nio-8080-exec-8] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception

org.springframework.security.web.firewall.RequestRejectedException: The request was rejected because the URL was not normalized.
	at org.springframework.security.web.firewall.StrictHttpFirewall.getFirewalledRequest(StrictHttpFirewall.java:248) ~[spring-security-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:193) ~[spring-security-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) ~[spring-security-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106) ~[spring-boot-actuator-1.5.13.RELEASE.jar!/:1.5.13.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]

With the self build of the current master of github it works. By using develop branch of engine, it is not working. Maybe I am forced to use the new api (which is hard without documentation)?

Does this error crash the container?

Also the current develop state is already relatively well documented, the only thing missing is the generated markdown rest api documentation. But you can always access the dynamic swagger page of the engine. (located at /swagger-ui.html).

[wurstbrot]

Does this error crash the container?

no