Current Issues | Ongoing Monitoring | Previous Issues | General Info

Current and Previous Issues affecting the Sandwell Schools Broadband Service

3/12/08 Netsweeper "Network Unavailable" problem

A number of schools reported problems with non-availability of websites that had previously been accessible. This appears to have been a configuration issue with the Netsweeper service. The netsweeper service is a fully managed filtering service, Netsweeper Inc. is a Canadian company. Netsweeper regularly make changes to the netsweeper policies to ensure that users are safeguarded. The nature of the internet is such that new threats appear daily and these may need to be addressed in different ways, i.e. using techniques other than adding entries to allow or deny lists. This appears to have been the cause of this problem and affected other Local Authorities as well as Sandwell. Synetrix are working with Netsweeper to resolve the problem.

(3/12/08 16.20:00): Update: Synetrix advise that the problem has been resolved.

(4/12/08, 11.30.00): Update: We are advised that the fault was caused by a failure of the Netsweeper Categorisation Servers, based in Canada, and affected all Netsweeper users worldwide. Broadband Sandwell apologises for any inconvenience that was caused.

11/11/2008 Cache Server Upgrade Completed, Service working normally

The work carried out last month to upgrade the caching servers (please see below) is now complete, and statistics that are being produced by the caches indicate that they appear to be working more efficiently than previously. In addition to this upgrade, the number of servers was increased so that schools' usage is spread over an increased number of caching servers.

Work is continuing on points 4 and 5 below, we will advise schools when this is complete.

17/10/2008 Message from Keith Chandler to all schools

Please accept my apologies for the continued problems that we are experiencing on the Broadband network.

Following the original problem, several work streams are underway:

1. A fault was identified that was causing a memory leak on the squids. The short term fix involves regular restarts of the squids (each night) and monitoring of available memory to manually restart when necessary

2. Synetrix is upgrading all of the squid servers and the Netsweeper policy servers to remove the problem. This upgrade is underway, the policy servers have been upgraded, and the squids are being upgraded in a rolling program. It was also determined early on that the communication method between the squids and the policy servers had changed, so it was necessary to retain an old policy server to talk to the old squids, and to deploy the new policy servers to talk to the new squids, adding to the complexity of the upgrade. This is almost complete, we had previously estimated that this would be complete by10th Oct, I am assured that it will be done by Friday 24th Oct.

3. A separate problem was identified with the IWSS AV servers concerning FTP downloads over HTTP. This is being fixed separately.

In addition:

4. We are negotiating with Transform Sandwell to upgrade the incoming ISP bandwidth from 100Mbit/s to 1Gbit/s. We expect this to be in place before the end of the academic year, this service will be shared with SMBC, but should allow a large increase in available inbound/outbound bandwidth.

5. In preparation for this upgrade, we have asked Synetrix to review the firewall configuration. Schools are currently connected through the firewalls using 100Mbit/s NICs on the firewalls themselves, these will be replaced with 1Gbit/s NICs shortly. Otherwise, no point having a 1Gbit/s feed if you can only use 100Mbit/s of it.

6. As well as this, Synetrix will be increasing the number of squids (and IWSSes) to take account of anticipated increase in bandwidth, which will inevitably lead to an increase in demand, so these too must be increased or else they won’t cope.

30/09/08 Ongoing slowness experienced by some schools

We are working with the ISP to bring a number of extra services on-line to address this problem. The following work, which will not be service-affecting, are scheduled:

• the commissioning of replacement netsweeper policy server following the recent (09/09/08) outage. This work is scheduled for completion by 8th October.
• creation of a separate non-AV'd service for use by schools on request and by negotiation. Details on request, completion by end October 2008
• Addition of extra monitoring tools to the network. This is scheduled for completion by end of November 2008.

09/09/08 Loss of Internet Service to All Sandwell Schools

Broadband Sandwell apologises to schools for problems that occurred on Tuesday 9th September 2008. A loss of service was identified at 08:35 on Tuesday morning and was immediately reported to the managed service provider, Synetrix Ltd. The problem was treated as the highest priority and a number of senior Synetrix engineers were involved in identifying the cause of the fault and its subsequent rectification. Unfortunately, it took some time for them to determine the cause which was eventually identified as being due to a faulty network interface that caused several other servers to fail. The faulty system was taken out of service at 3.00pm and will be repaired.

Broadband Sandwell are in discussion with Synetrix to try to determine why the cause of the issue was not identified more quickly and to ensure that their monitoring systems are upgraded so that any subsequent similar faults are resolved more quickly.

Until the faulty server is repaired and returned to service, there may be occasional "drop outs" due to the remaining servers being placed under additional stress. Schools are asked to re-try any failed web browsing attempts by revisiting the website concerned and report persistent problems to our helpdesk.

09/06/08 Sandwell MBC Power Testing

Sandwell MBC advise us that they plan to carry out some tests of the recent electrical upgrade on Sunday 29th June 2008 which may result in loss of some services during the day.

As with previous power down issues, Broadband Sandwell will manage the powerdown cycle to minimise the possibility of problems on Monday 30th June.

We additionally plan to carry out some maintenance work on the Webmail system to try to resolve the disk space issue (see previous issue)

26/06/08 Webmail Unavailability

Schools experienced problems with the Webmail service at the end of May. These were caused by a number of large emails being sent to a number of webmail users, resulting in all disk space being used to store these emails.

The problem is a recurrence of a previous problem. Click here for more details

25/03/08 Sandwell Extranet Problems

Schools are advised that problems accessing the Sandwell Extranet website have now been rectified. Please note that this was a Sandwell MBC issue and NOT a Schools' Broadband Network fault.

28/02/08 Internet Connection Problems

Schools across the borough are currently experiencing internet outage. This was reported to the contractor, Synetrix Ltd, on 27/02/08 and immediately escalated. A "work around" was put into place late on Wednesday evening but failed again this morning.

28.02.08 9:45am Synetrix advise that the service is now restored. The problem has been identified as a faulty DNS server, which has been removed. Synetrix will be carrying out further testing to determine the root cause of the problem.

21/02/08 Webmail Digital Certificates

Sandwell Council advise us that the Webmail service will require new digital certificates from May 2008. Users who may have installed the old certificates (so that you don't have to click OK to the certificate warning each time you log on) will have to remove the old certificate and replace it with the new one following the upgrade. Contact us if you have difficulty once the upgrade has taken place.

31/11/07

Transform Sandwell advise us that they are planning to make changes to the IP addresses of their web services. This will result in schools websites and webmail services being unavailable over the weekend  of 15th- 16th December.

This will have the following implications for schools:

Over the weekend of 15th-16th December:

• SMBC Webmail will be unavailable
• schools websites for most schools (where the domain name ends with .sandwell.sch.uk) will be unavailable
• delivery of email to all .sandwell.sch.uk domains may be delayed
• Other SMBC services, such as SBS, Atrium and BMS will be unavailable
• Other Broadband Sandwell services will NOT be affected

After the weekend:

• changes may be needed to DNS services for some schools. This will be undertaken by Broadband Sandwell, but may involve fault finding to resolve some problems. Please report any faults to the Broadband Sandwell helpdesk in the usual way
• if schools connect to Sandwell services using IP information rather than by domain name, configurations will need to be changed. This may affect SFTP access to Sandwell webservers if your technician uses the ip address rather than the domain name. If you are affected by this, click here for more information
• It is expected that after the weekend, all Sandwell Council services will operate normally. Any exceptions will be caused by local re-configuration requirements

22/10/07

Please note that Menzies High School will have no power in school on Monday 22nd October between the hours of 8.00am and 11:00am. Also Thursday 25th October between the hours of 8.00am - 2.00pm.

27/08/07

Sandwell MBC advise that they need to carry out further work on the power supply to the server room at Oldbury Council House on Sunday, 30th September 2007. Although not related to the schools' broadband service, this will necessitate a loss of the broadband service during that day.

• This has resulted in the replacement of the old fuse boards (these were large floor standing units separately wired for each circuit, requiring down time each time a new circuit was added), with a pair of new type enclosures, having external "pluggable" commando-type connectors, such that a new circuit can simply be plugged in. It is expected that this will cut down the amount of down time needed in the future.

24/07/07

Synetrix have requested a service interruption on 31st July 2007 for essential maintenance work. This work "follows on" from a problem uncovered during the power outage on 15th July 2007, when a faulty module was discovered. Replacement of this equipment is scheduled for 18:00hrs on 31st July 2007, and will last for up to 4 hours.

• When the module was replaced it was discovered that there was a problem with the frame into which it plugged. This was subsequently replaced under the maintenance agreement.

13/07/07: We are advised that the power issue at Menzies High School is now resolved.

• The school undertook a fuse board upgrade in the school which was "tripping out"

03/07/07: Sandwell MBC advise us that there is planned work to be carried out at Oldbury Council House on Sunday 15th July 2007 to update power supplies in the Council House. Although not related to the schools' broadband service, this will necessitate a loss of the broadband service during that day.

• This is to allow the addition of extra circuits for new computer systems to be installed in the Sandwell Council Server room.

27/06/07: There are currently power issues at Menzies High School. This will affect the schools who get their internet feed through via Menzies High. Extensive work is scheduled to be carried out to resolve this issue out of hours.

• see above

March 07:

Improving the Internet feed

Following testing, the Sandell Schools Broadband network is now using the UKERNA ISP feed

Click here for more information

March 07: Loss of Service

On Tuesday 13th March a loss of service was experienced. This was resolved by Synetrix within 1 and a half hours of notification which complies with the 4 hour SLA resolution time for a Priority 1 fault.

Dec 06: ISDN Connections

We are advised that some schools are still paying for the old "Schools Internet Caller" service which was made redundant by the connection to the Broadband Sandwell network. Please check that you are not still paying unnecessary bills.

iptest