Current Issues | Ongoing Monitoring | Previous Issues | General Info

Previous Issues affecting the Sandwell Schools Broadband Service

02/08/2011 Planned Loss of service w/c 15th August and 22nd August 2011

Please see Sandwell MBC Extranet Circular 161 for further details of the planned outage to replace core switches.

http://www.lea.sandwell.gov.uk/members/bulletin/circulars/2011/docs/161.pdf

*********** update, 15th August 2011***********

The first phase of this work, to replace the core switch at Sandwell Council House, Oldbury, was completed successfully. The next scheduled work, to replace the core switch at Sandwell Hospital, will take place on Friday 19th August. This will affect schools in the Tipton, Wednesbury and West Bromwich areas.

*********** update, 19th August 2011***********

The second phase of this work, to replace the core switch at Sandwell Hospital, West Bromwich, was completed successfully. The next scheduled work, to replace the core switch at Smethwick Council House, Smethwick, will take place on Monday 22nd August. This will affect schools in the Smethwick and Oldbury areas.

*********** update, 22nd August 2011***********

The third phase of this work, to replace the core switch at Smethwick Town Hall, was completed successfully. The next scheduled work, to complete the configuration and improve resilience, will take place on Friday 26nd August. There may be short periods of unavailability during the day but the engineers will make efforts to avoid this.

*********** update, 26th August 2011***********

The final phase of this work, to to complete the configuration and improve resilience, was completed successfully. The upgrade is complete.

**************************************************

29/07/2011 Loss of service for schools served from the St Michaels node

We have corresponded separately to affected schools. Following the St Michaels move to the new Rowley BSF site, the old school has been decommissioned and services removed. Unfortunately, circuits provided by Virgin Media have taken longer than expected to remove and the old service was decommissioned on Friday 29th July. This was also the expected swapover date for Virgin Media, and most of the work was completed such that service was restored to most of the affected schools by end of the day, with a further two schools reconnected Monday am.

An unanticipated problem with access to a pit in a road in the Rowley area means that four of the schools are still waiting for service to be restored. We are pressing Virgin Media for an early date for this to be completed.

This work was completed during the week of 1st August, service to all affected schools is now restored.

17/03/2011 Loss of Internet connection between 10.30pm and 8.30am on 18/03/2011

In the evening of 17/03/2011, out of hours work was being undertaken to correct a problem with the Internet feed. Unfortunately, once complete, access to the service was lost until early in the morning when an engineer was able to gain access. Broadband Sandwell apologise for the loss of service The underlying work, to correct a slowness problem affecting some schools, is being monitored to determine whether the changes made have resolved the issue.

15/03/2011 Loss of Internet connection between 2.00pm and 3.00pm

We have been advised by Synetrix that this outage was caused by a DNS change upstream of the schools network. This change was not expected to have any effect on the Sandwell network but unfortunately did. Broadband Sandwell apologise for the loss of service at that time.

16/11/2010 Advance Notice of of network unavailability on Sunday 21st November 2010 between 8.00am and 6.00pm

We have been advised by Transform Sandwell that the electrical power in Oldbury Council House will be turned off on Sunday 21st November 2010 for urgent maintenance work.

This is to repair a system at Oldbury Council House that recently failed to start up the backup generators during an external power outage.

This will affect the schools’ broadband network and school websites on that day.

It is anticipated that the power will be off from 8am on Sunday morning until 6pm Sunday afternoon. As a precaution, some services will be taken off-line on Friday 19th pm, and may not be restored until the morning of Monday 22nd, although the broadband itself will only be affected on Sunday. Broadband Sandwell apologises for any inconvenience that may be caused.

8/9/2010 Interruption to service for schools in Tipton and Wednesbury areas

Broadband Sandwell apologises to users in the Tipton and Wednesbury areas for disruption that occurred on Tuesday 8th September 2010. This was caused by power problems at Wood Green High School, which is a major node on our network. A temporary power supply has been established to bypass a faulty UPS at the school. It is expected that once the UPS is repaired, the service will need to be taken off line again to restore the equipment to its usual configuration. We anticipate carrying out this work out of hours to minimise further disruption. (Please note that the faulty equipment is part of the broadband network, and not the responsibility of the school.)

16/07/2010 Disruption to the Smethwick node

Broadband Sandwell apologises to users in the Smethwick area for disruption that occurred on Friday 16th July 2010. This was caused by an intruder who, allegedly high on drink or drugs, broke in to the Smethwick Council buildings in the early hours. Having smashed a window to climb in (and cutting himself badly) this individual managed to get in to the network room at the building, where he was later found. At some point he unplugged a number of connections in the schools network cabinets and damaged some equipment. Staff from Broadband Sandwell attended the site as soon as we were aware of the problem, but access was restricted because of spilt blood and the police investigation. Repairs were made difficult by the multiple faults and restrictions imposed by the incident. Most schools were reconnected by mid afternoon.

23/06/2010 reports of slow intenet

We have this logged with Synetrix, however... we believe that it may relate to the fact that there is a football match on this afternoon and people may be trying to view this over the broadband connection. If so, this may not be as successful as users may hope; there will be others trying to do the same thing, and considerations should be given to the streaming servers at the remote end (ITV website) which may clog up because there are too many connections. It may be better to use a television source to watch the match, by routing a TV aerial signal through a video recorder to your projector or large screen. Please consider that the TV network is designed for broadcasting to large numbers, whereas the internet is caters better for small audiences and larger variety. Please note also that whether you use a PC or a televison, there will be licensing issues, particularly if the event is shown to a large audience. Please visit the TV Licensing website for more details.

09/06/2010 Advance Notice of of network unavailability on Sunday 20th June 2010 between 8.00am and 6.00pm

We have been advised by Transform Sandwell that the electrical power in Oldbury Council House will be turned off on Sunday 20th June 2010 for urgent electrical work.

This will be to implement a “power perfector device” that resides between the electrical feed to both inside and outside Sandwell Council House.

This will affect the schools’ broadband network and school websites on that day.

It is anticipated that the power will be off from 8am on Sunday morning until 6pm Sunday afternoon. As a precaution, some services will be taken off-line on Friday 18th pm, and may not be restored until the morning of Monday 21st June. Broadband Sandwell apologises for any inconvenience that may be caused.

22/07/2009 Advance Notice of network unavailability on Sunday 2nd August 2009 between 9.00am and 5.00pm

Transform Sandwell advise that essential maintenance work in the Council House on Sunday 2nd August will result in loss of power to all services, including the schools' broadband service. The service will be available for use on Monday, 3rd August. Please call Broadband Sandwell if this will cause problems with school activities.

8/06/2009 Advance notice of network unavailability on Saturday 20th June 2009 between 10am and 10.30am

Unfortunately, it was found that the maintenance period on May 15th (see below) was not sufficient to allow the work to be completed, and so we are advised that another short maintenance period for the upgrade of the Shireland Learning Gateway service is planned for Saturday 20th June at 10.00am.

This may cause loss of broadband service to schools for up to 30 minutes.

Please call Broadband Sandwell if this will cause problems with school activities.

2/06/2009: Unavailability of broadband services between 2.30pm and 5.30pm today

Broadband Sandwell apologise for this unexpected loss of service, which was caused by a power failure in the Sandwell Council House Server room. The fault only affected parts of the service as it was rectified before all of the UPS backup electrical supply was exhausted. The fault is being investigated by Sandwell MBC and Transform Sandwell to prevent a recurrence of this incident.

15/05/2009 Advance notice of network unavailability on Saturday 16th May 2009 between 10am and 10.30am

We are advised that maintenance work for the Shireland Learning Gateway may cause loss of broadband service to schools on Saturday for up to 30 minutes. Broadband Sandwell apologise for the short notice.

Please call Broadband Sandwell if this will cause problems with school activities.

25/03/2009 Advance notice of network unavailability on Thursday 16th and Friday 17th April 2009

As part of the previously advised upgrade of the Sandwell Schools Broadband Network which will allow greater bandwidth availability to schools, a replacement firewall has been procured for deployment into the network's server array. Replacement of the current firewall with the newly procured device will unfortunately involve loss of service during the changeover. This is planned to take place on the 16th and 17th April (the Thursday and Friday of week 2 of the Easter holiday). Internet connection will be unavailable at that time. If the proposed changeover is not successful, the system will be returned to the previous state to avoid problems the following Monday.

Please call Broadband Sandwell if this will cause problems with school activities.

29/01/2009 Advance notice of network unavailability on Friday 6th March 2009

Readers of this page will recall that an earlier posting (17/10/2008, see below) included information about a forthcoming upgrade to the broadband connection to allow greater bandwidth availability to schools. This will include essential firewall changes. Work in this area has been progressing since that time, and the first of these changes has been scheduled to take place on Friday 6th March 2009, between 4pm and 8pm. The change will unavoidably cause loss of connection during this time.

Once the changes have been made, testing will be carried out to confirm that the new configuration performs correctly. If the proposed changes are not successful, the system will be returned to the previous state to avoid problems the following monday.

Please call Broadband Sandwell if this will cause problems with school activities.

3/12/08 Netsweeper "Network Unavailable" problem

A number of schools reported problems with non-availability of websites that had previously been accessible. This appears to have been a configuration issue with the Netsweeper service. The netsweeper service is a fully managed filtering service, Netsweeper Inc. is a Canadian company. Netsweeper regularly make changes to the netsweeper policies to ensure that users are safeguarded. The nature of the internet is such that new threats appear daily and these may need to be addressed in different ways, i.e. using techniques other than adding entries to allow or deny lists. This appears to have been the cause of this problem and affected other Local Authorities as well as Sandwell. Synetrix are working with Netsweeper to resolve the problem.

(3/12/08 16.20:00): Update: Synetrix advise that the problem has been resolved.

(4/12/08, 11.30.00): Update: We are advised that the fault was caused by a failure of the Netsweeper Categorisation Servers, based in Canada, and affected all Netsweeper users worldwide. Broadband Sandwell apologises for any inconvenience that was caused.

11/11/2008 Cache Server Upgrade Completed, Service working normally

The work carried out last month to upgrade the caching servers (please see below) is now complete, and statistics that are being produced by the caches indicate that they appear to be working more efficiently than previously. In addition to this upgrade, the number of servers was increased so that schools' usage is spread over an increased number of caching servers.

Work is continuing on points 4 and 5 below, we will advise schools when this is complete.

17/10/2008 Message from Keith Chandler to all schools

Please accept my apologies for the continued problems that we are experiencing on the Broadband network.

Following the original problem, several work streams are underway:

1. A fault was identified that was causing a memory leak on the squids. The short term fix involves regular restarts of the squids (each night) and monitoring of available memory to manually restart when necessary

2. Synetrix is upgrading all of the squid servers and the Netsweeper policy servers to remove the problem. This upgrade is underway, the policy servers have been upgraded, and the squids are being upgraded in a rolling program. It was also determined early on that the communication method between the squids and the policy servers had changed, so it was necessary to retain an old policy server to talk to the old squids, and to deploy the new policy servers to talk to the new squids, adding to the complexity of the upgrade. This is almost complete, we had previously estimated that this would be complete by10th Oct, I am assured that it will be done by Friday 24th Oct.

3. A separate problem was identified with the IWSS AV servers concerning FTP downloads over HTTP. This is being fixed separately.

In addition:

4. We are negotiating with Transform Sandwell to upgrade the incoming ISP bandwidth from 100Mbit/s to 1Gbit/s. We expect this to be in place before the end of the academic year, this service will be shared with SMBC, but should allow a large increase in available inbound/outbound bandwidth.

5. In preparation for this upgrade, we have asked Synetrix to review the firewall configuration. Schools are currently connected through the firewalls using 100Mbit/s NICs on the firewalls themselves, these will be replaced with 1Gbit/s NICs shortly. Otherwise, no point having a 1Gbit/s feed if you can only use 100Mbit/s of it.

6. As well as this, Synetrix will be increasing the number of squids (and IWSSes) to take account of anticipated increase in bandwidth, which will inevitably lead to an increase in demand, so these too must be increased or else they won’t cope.

30/09/08 Ongoing slowness experienced by some schools

We are working with the ISP to bring a number of extra services on-line to address this problem. The following work, which will not be service-affecting, are scheduled:

• the commissioning of replacement netsweeper policy server following the recent (09/09/08) outage. This work is scheduled for completion by 8th October.
• creation of a separate non-AV'd service for use by schools on request and by negotiation. Details on request, completion by end October 2008
• Addition of extra monitoring tools to the network. This is scheduled for completion by end of November 2008.

09/09/08 Loss of Internet Service to All Sandwell Schools

Broadband Sandwell apologises to schools for problems that occurred on Tuesday 9th September 2008. A loss of service was identified at 08:35 on Tuesday morning and was immediately reported to the managed service provider, Synetrix Ltd. The problem was treated as the highest priority and a number of senior Synetrix engineers were involved in identifying the cause of the fault and its subsequent rectification. Unfortunately, it took some time for them to determine the cause which was eventually identified as being due to a faulty network interface that caused several other servers to fail. The faulty system was taken out of service at 3.00pm and will be repaired.

Broadband Sandwell are in discussion with Synetrix to try to determine why the cause of the issue was not identified more quickly and to ensure that their monitoring systems are upgraded so that any subsequent similar faults are resolved more quickly.

Until the faulty server is repaired and returned to service, there may be occasional "drop outs" due to the remaining servers being placed under additional stress. Schools are asked to re-try any failed web browsing attempts by revisiting the website concerned and report persistent problems to our helpdesk.

09/06/08 Sandwell MBC Power Testing

Sandwell MBC advise us that they plan to carry out some tests of the recent electrical upgrade on Sunday 29th June 2008 which may result in loss of some services during the day.

As with previous power down issues, Broadband Sandwell will manage the powerdown cycle to minimise the possibility of problems on Monday 30th June.

We additionally plan to carry out some maintenance work on the Webmail system to try to resolve the disk space issue (see previous issue)

26/06/08 Webmail Unavailability

Schools experienced problems with the Webmail service at the end of May. These were caused by a number of large emails being sent to a number of webmail users, resulting in all disk space being used to store these emails.

The problem is a recurrence of a previous problem. Click here for more details

25/03/08 Sandwell Extranet Problems

Schools are advised that problems accessing the Sandwell Extranet website have now been rectified. Please note that this was a Sandwell MBC issue and NOT a Schools' Broadband Network fault.

28/02/08 Internet Connection Problems

Schools across the borough are currently experiencing internet outage. This was reported to the contractor, Synetrix Ltd, on 27/02/08 and immediately escalated. A "work around" was put into place late on Wednesday evening but failed again this morning.

28.02.08 9:45am Synetrix advise that the service is now restored. The problem has been identified as a faulty DNS server, which has been removed. Synetrix will be carrying out further testing to determine the root cause of the problem.

21/02/08 Webmail Digital Certificates

Sandwell Council advise us that the Webmail service will require new digital certificates from May 2008. Users who may have installed the old certificates (so that you don't have to click OK to the certificate warning each time you log on) will have to remove the old certificate and replace it with the new one following the upgrade. Contact us if you have difficulty once the upgrade has taken place.

31/11/07

Transform Sandwell advise us that they are planning to make changes to the IP addresses of their web services. This will result in schools websites and webmail services being unavailable over the weekend  of 15th- 16th December.

This will have the following implications for schools:

Over the weekend of 15th-16th December:

• SMBC Webmail will be unavailable
• schools websites for most schools (where the domain name ends with .sandwell.sch.uk) will be unavailable
• delivery of email to all .sandwell.sch.uk domains may be delayed
• Other SMBC services, such as SBS, Atrium and BMS will be unavailable
• Other Broadband Sandwell services will NOT be affected

After the weekend:

• changes may be needed to DNS services for some schools. This will be undertaken by Broadband Sandwell, but may involve fault finding to resolve some problems. Please report any faults to the Broadband Sandwell helpdesk in the usual way
• if schools connect to Sandwell services using IP information rather than by domain name, configurations will need to be changed. This may affect SFTP access to Sandwell webservers if your technician uses the ip address rather than the domain name. If you are affected by this, click here for more information
• It is expected that after the weekend, all Sandwell Council services will operate normally. Any exceptions will be caused by local re-configuration requirements

22/10/07

Please note that Menzies High School will have no power in school on Monday 22nd October between the hours of 8.00am and 11:00am. Also Thursday 25th October between the hours of 8.00am - 2.00pm.

27/08/07

Sandwell MBC advise that they need to carry out further work on the power supply to the server room at Oldbury Council House on Sunday, 30th September 2007. Although not related to the schools' broadband service, this will necessitate a loss of the broadband service during that day.

• This has resulted in the replacement of the old fuse boards (these were large floor standing units separately wired for each circuit, requiring down time each time a new circuit was added), with a pair of new type enclosures, having external "pluggable" commando-type connectors, such that a new circuit can simply be plugged in. It is expected that this will cut down the amount of down time needed in the future.

24/07/07

Synetrix have requested a service interruption on 31st July 2007 for essential maintenance work. This work "follows on" from a problem uncovered during the power outage on 15th July 2007, when a faulty module was discovered. Replacement of this equipment is scheduled for 18:00hrs on 31st July 2007, and will last for up to 4 hours.

• When the module was replaced it was discovered that there was a problem with the frame into which it plugged. This was subsequently replaced under the maintenance agreement.

13/07/07: We are advised that the power issue at Menzies High School is now resolved.

• The school undertook a fuse board upgrade in the school which was "tripping out"

03/07/07: Sandwell MBC advise us that there is planned work to be carried out at Oldbury Council House on Sunday 15th July 2007 to update power supplies in the Council House. Although not related to the schools' broadband service, this will necessitate a loss of the broadband service during that day.

• This is to allow the addition of extra circuits for new computer systems to be installed in the Sandwell Council Server room.

27/06/07: There are currently power issues at Menzies High School. This will affect the schools who get their internet feed through via Menzies High. Extensive work is scheduled to be carried out to resolve this issue out of hours.

• see above

March 07:

Improving the Internet feed

Following testing, the Sandell Schools Broadband network is now using the UKERNA ISP feed

Click here for more information

March 07: Loss of Service

On Tuesday 13th March a loss of service was experienced. This was resolved by Synetrix within 1 and a half hours of notification which complies with the 4 hour SLA resolution time for a Priority 1 fault.

Dec 06: ISDN Connections

We are advised that some schools are still paying for the old "Schools Internet Caller" service which was made redundant by the connection to the Broadband Sandwell network. Please check that you are not still paying unnecessary bills.

iptest