I believe you would need to write a Java-based OperationAdvisor for validating the time.  Within your OperationAdvisor, you would check something like this (just pseudo-code, not real code)
      if (type == task) {
           if (effort > 8 hours)
                add warning message to Collector and return
      }
      else if (type == story)  {
           get all the children and iterate over them, adding up the efforts of all the children
           if (totalEffort > 40 hours               
                  add warning message to Collector and return
      }

If you do it this way, the TotalStory warning would show on the Story when saved, *not* on the child task when saved.  If you want the warning for the TotalStory hours to be on the child task when it is saved (like ... you are saving a task which may be okay by itself, but when added to the parent effort, exceeds 40 hours) .. then you would have to change the OperationAdvisor to have everything with the Task block, and for the task, go get it's parent, then go get the children and iterate over them.

How you report on that .. well ... I don't know that one :-)

Susan

In general, you can use Attribute Customization - JavaScript and Java validators to do things like that. See https://jazz.net/wiki/bin/view/Main/AttributeCustomization#Validators and https://jazz.net/library/article/1093 Lab 5 .

JavaScript is limited to accessing attributes of the work Item. As far as I am aware the rollup and planning data you can see in work items are not an attribute, but a special presentation. Therefore, as Susan mentions, you would have to use a java based validator as described in https://rsjazz.wordpress.com/2013/06/26/attribute-customization-java-based-value-providers-conditions-and-validators/

You would have to calculate the rollup yourself, or use some internal API, if you can find it, because planning does not expose its API. https://rsjazz.wordpress.com/?s=links provides several posts on how to follow links to do the rollup.